2024-01-21T10:45:09.282 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T10:45:09.282 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441 --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/orig.config.yaml 2024-01-21T10:45:09.330 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T10:45:09.419 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7524441' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8186 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdWB1nsnue4egBQNIScTUTCvwAOSIXqEtH4of5ooKpUC2LgMPZhxI76S7i84nd+YuOCjGEXDK/d8duzbvI9woM= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFK1cxZMOuJXvdboXT73THFXxhxywny3PuE2Tyt3iXmqc/MURC8FjKuX8T1gMjkPk85OXlglO27FoE5++VH1Jok= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_23:17:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T10:45:09.419 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T10:45:09.419 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T10:45:09.420 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T10:45:09.421 INFO:teuthology.task.internal:Checking packages... 2024-01-21T10:45:09.443 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T10:45:09.444 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T10:45:09.444 INFO:teuthology.packaging:ref: None 2024-01-21T10:45:09.444 INFO:teuthology.packaging:tag: None 2024-01-21T10:45:09.444 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T10:45:09.444 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T10:45:09.444 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T10:45:09.669 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T10:45:09.671 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T10:45:09.678 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T10:45:09.678 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T10:45:09.720 INFO:teuthology.task.internal:Saving configuration 2024-01-21T10:45:09.737 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T10:45:09.743 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T10:45:09.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 10:39:48.591383', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdWB1nsnue4egBQNIScTUTCvwAOSIXqEtH4of5ooKpUC2LgMPZhxI76S7i84nd+YuOCjGEXDK/d8duzbvI9woM='} 2024-01-21T10:45:09.787 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 10:39:48.593264', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFK1cxZMOuJXvdboXT73THFXxhxywny3PuE2Tyt3iXmqc/MURC8FjKuX8T1gMjkPk85OXlglO27FoE5++VH1Jok='} 2024-01-21T10:45:09.787 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T10:45:09.793 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T10:45:09.793 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T10:45:09.794 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T10:45:09.879 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbef173e160>, signals=[15]) 2024-01-21T10:45:09.879 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T10:45:09.885 INFO:teuthology.task.internal:Opening connections... 2024-01-21T10:45:09.885 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-01-21T10:45:09.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:45:09.962 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-01-21T10:45:09.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:45:10.033 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T10:45:10.039 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-01-21T10:45:10.061 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-01-21T10:45:10.061 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-01-21T10:45:10.122 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-01-21T10:45:10.122 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="8" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="8" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el8" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T10:45:10.123 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T10:45:10.124 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-01-21T10:45:10.149 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-01-21T10:45:10.173 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-01-21T10:45:10.173 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-01-21T10:45:10.235 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-01-21T10:45:10.235 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T10:45:10.236 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T10:45:10.237 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-01-21T10:45:10.257 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T10:45:10.266 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T10:45:10.286 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T10:45:10.286 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-01-21T10:45:10.288 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-01-21T10:45:10.309 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T10:45:10.314 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T10:45:10.314 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-01-21T10:45:10.348 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-01-21T10:45:10.375 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T10:45:10.431 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T10:45:10.445 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T10:45:10.445 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T10:45:10.446 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T10:45:10.446 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T10:45:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T10:45:10.466 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-01-21T10:45:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T10:45:10.467 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-01-21T10:45:10.526 INFO:teuthology.orchestra.run.smithi089.stdout:4.18.0-489.el8.x86_64 2024-01-21T10:45:10.526 INFO:teuthology.task.kernel:Running kernel on smithi089: 4.18.0-489.el8.x86_64 2024-01-21T10:45:10.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-01-21T10:45:10.527 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-489.el8.x86_64 2024-01-21T10:45:10.527 INFO:teuthology.task.kernel:Running kernel on smithi178: 4.18.0-489.el8.x86_64 2024-01-21T10:45:10.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2024-01-21T10:46:29.021 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - AppStream 356 kB/s | 26 MB 01:15 2024-01-21T10:46:34.870 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - AppStream 330 kB/s | 26 MB 01:21 2024-01-21T10:47:11.958 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - BaseOS 275 kB/s | 10 MB 00:38 2024-01-21T10:47:14.008 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras 39 kB/s | 18 kB 00:00 2024-01-21T10:47:14.400 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 6.9 kB 00:00 2024-01-21T10:47:14.652 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-21T10:47:16.422 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 16 MB 00:01 2024-01-21T10:47:17.197 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - BaseOS 279 kB/s | 10 MB 00:37 2024-01-21T10:47:19.080 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras 63 kB/s | 18 kB 00:00 2024-01-21T10:47:19.331 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras common packages 48 kB/s | 6.9 kB 00:00 2024-01-21T10:47:19.583 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-21T10:47:20.153 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 376 kB/s | 24 kB 00:00 2024-01-21T10:47:20.961 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-01-21T10:47:23.041 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T10:47:23.041 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T10:47:23.158 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T10:47:23.159 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T10:47:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T10:47:23.159 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T10:47:23.159 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T10:47:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout:Install 3 Packages 2024-01-21T10:47:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T10:47:23.161 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 89 M 2024-01-21T10:47:23.161 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 96 M 2024-01-21T10:47:23.161 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T10:47:24.775 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 294 kB/s | 24 kB 00:00 2024-01-21T10:47:27.696 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T10:47:27.697 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T10:47:27.812 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T10:47:27.813 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T10:47:27.813 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T10:47:27.813 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T10:47:27.813 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T10:47:27.813 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout:Install 3 Packages 2024-01-21T10:47:27.814 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T10:47:27.815 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 89 M 2024-01-21T10:47:27.815 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 96 M 2024-01-21T10:47:27.815 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T10:48:37.954 INFO:teuthology.orchestra.run.smithi089.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 140 kB/s | 10 MB 01:14 2024-01-21T10:49:17.909 INFO:teuthology.orchestra.run.smithi178.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 95 kB/s | 10 MB 01:50 2024-01-21T10:51:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 141 kB/s | 36 MB 04:20 2024-01-21T10:52:25.803 INFO:teuthology.orchestra.run.smithi178.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 123 kB/s | 36 MB 04:57 2024-01-21T10:52:27.740 INFO:teuthology.orchestra.run.smithi089.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 145 kB/s | 43 MB 05:04 2024-01-21T10:52:27.740 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T10:52:27.740 INFO:teuthology.orchestra.run.smithi089.stdout:Total 300 kB/s | 89 MB 05:04 2024-01-21T10:52:28.411 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T10:52:28.833 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T10:52:28.833 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T10:52:29.228 INFO:teuthology.orchestra.run.smithi178.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 146 kB/s | 43 MB 05:01 2024-01-21T10:52:29.228 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T10:52:29.228 INFO:teuthology.orchestra.run.smithi178.stdout:Total 303 kB/s | 89 MB 05:01 2024-01-21T10:52:29.905 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T10:52:30.071 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T10:52:30.095 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T10:52:30.337 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T10:52:30.337 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T10:52:31.574 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T10:52:31.600 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T10:52:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T10:52:33.488 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T10:52:34.408 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T10:52:34.913 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T10:52:37.061 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T10:52:37.479 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T10:52:38.489 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T10:52:38.921 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T10:52:40.904 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T10:52:41.168 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:52:42.340 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T10:52:42.592 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:03.082 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T10:53:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T10:53:03.928 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:03.929 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T10:53:03.929 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T10:53:03.929 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T10:53:03.929 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T10:53:03.929 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T10:53:03.929 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T10:53:04.093 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T10:53:04.313 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T10:53:04.314 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T10:53:04.314 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T10:53:04.314 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T10:53:04.314 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T10:53:04.314 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T10:53:04.314 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T10:53:04.314 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T10:53:04.315 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:04.336 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:04.523 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T10:53:04.524 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T10:53:04.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-01-21T10:53:04.557 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.02 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 129.el8 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Group : System Environment/Base 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 9265586 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T10:53:04.558 INFO:teuthology.orchestra.run.smithi089.stdout:Relocations : (not relocatable) 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : CentOS BuildSystem 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T10:53:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T10:53:04.560 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T10:53:04.560 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 4.18.0-535.el8.x86_64 2024-01-21T10:53:04.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T10:53:04.685 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:04.686 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T10:53:04.686 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T10:53:05.151 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T10:53:05.151 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T10:53:05.151 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T10:53:05.152 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T10:53:05.152 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T10:53:05.152 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T10:53:05.152 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T10:53:05.294 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T10:53:05.515 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T10:53:05.515 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T10:53:05.515 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T10:53:05.515 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T10:53:05.516 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T10:53:05.516 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T10:53:05.516 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T10:53:05.516 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T10:53:05.725 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T10:53:05.726 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T10:53:05.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2024-01-21T10:53:05.759 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2024-01-21T10:53:05.759 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.02 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 129.el8 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Group : System Environment/Base 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 9265586 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Relocations : (not relocatable) 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : CentOS BuildSystem 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2024-01-21T10:53:05.760 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T10:53:05.761 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T10:53:05.762 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T10:53:05.762 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 4.18.0-535.el8.x86_64 2024-01-21T10:53:05.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T10:53:06.405 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-01-21T10:53:07.460 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-01-21T10:53:08.739 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-01-21T10:53:08.742 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-01-21T10:53:08.816 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.kRjEildjQU 2024-01-21T10:53:08.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kRjEildjQU 2024-01-21T10:53:08.890 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.kRjEildjQU 2024-01-21T10:53:09.047 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.kRjEildjQU is 6KB 2024-01-21T10:53:09.095 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.kRjEildjQU 2024-01-21T10:53:09.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T10:53:09.185 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T10:53:09.185 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T10:53:09.185 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T10:53:09.185 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T10:53:09.186 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T10:53:09.312 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-01-21T10:53:09.314 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-01-21T10:53:09.349 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.sKa1AXTgGe 2024-01-21T10:53:09.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sKa1AXTgGe 2024-01-21T10:53:09.393 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-01-21T10:53:09.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.sKa1AXTgGe 2024-01-21T10:53:09.560 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.sKa1AXTgGe is 6KB 2024-01-21T10:53:09.608 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.sKa1AXTgGe 2024-01-21T10:53:09.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T10:53:09.696 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T10:53:09.696 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T10:53:09.697 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T10:53:09.697 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T10:53:09.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T10:53:09.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-01-21T10:53:39.408 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T10:53:39.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-01-21T10:53:39.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:53:39.877 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T10:53:39.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T10:53:39.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:53:57.948 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-01-21T10:53:58.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-01-21T10:54:06.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-01-21T10:54:06.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:54:07.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T10:54:07.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:54:10.013 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-01-21T10:54:10.272 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-01-21T10:54:22.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-01-21T10:54:22.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:54:22.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T10:54:22.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:54:54.112 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-01-21T10:54:55.161 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-01-21T10:54:55.161 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T10:54:55.162 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-01-21T10:54:55.659 INFO:teuthology.orchestra.run.smithi089.stdout:4.18.0-535.el8.x86_64 2024-01-21T10:54:55.659 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T10:54:55.659 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T10:54:55.659 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T10:54:56.659 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T10:54:56.660 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T10:54:56.726 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-01-21T10:54:56.740 DEBUG:teuthology.parallel:result is None 2024-01-21T10:55:22.275 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T10:55:37.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T10:55:37.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T10:55:37.583 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-01-21T10:55:37.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T10:55:37.877 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T10:55:37.878 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-01-21T10:55:37.897 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-535.el8.x86_64 2024-01-21T10:55:37.897 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T10:55:37.897 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T10:55:37.898 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T10:55:38.898 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T10:55:38.898 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T10:55:38.971 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-01-21T10:55:38.985 DEBUG:teuthology.parallel:result is None 2024-01-21T10:55:38.985 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T10:55:38.993 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T10:55:38.993 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T10:55:38.997 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T10:55:39.017 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T10:55:39.028 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T10:55:39.070 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T10:55:39.070 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T10:55:39.073 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T10:55:39.110 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T10:55:39.138 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T10:55:39.139 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T10:55:39.142 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T10:55:39.176 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T10:55:39.185 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T10:55:39.186 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T10:55:39.195 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T10:55:39.197 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T10:55:39.203 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T10:55:39.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T10:55:39.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T10:55:39.266 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T10:55:39.274 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T10:55:39.274 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T10:55:39.300 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T10:55:39.324 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T10:55:39.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T10:55:39.453 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T10:55:39.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T10:55:39.579 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T10:55:39.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T10:55:39.645 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T10:55:39.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T10:55:39.746 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T10:55:39.831 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T10:55:39.875 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T10:55:39.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T10:55:39.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-01-21T10:55:39.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-01-21T10:55:40.007 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T10:55:40.045 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T10:55:40.445 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T10:55:40.453 INFO:teuthology.task.internal:Starting timer... 2024-01-21T10:55:40.453 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T10:55:40.498 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T10:55:40.596 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T10:55:40.596 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-01-21T10:55:40.678 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-01-21T10:55:40.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-01-21T10:55:40.744 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-01-21T10:55:40.746 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T10:55:40.746 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-01-21T10:55:40.798 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-01-21T10:55:40.798 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-01-21T10:55:40.835 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-01-21T10:55:40.835 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-01-21T10:55:40.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T10:55:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T10:55:40.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T10:55:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T10:55:40.903 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T10:55:40.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-01-21T10:55:40.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-01-21T10:55:40.970 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T10:55:40.980 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-21T10:55:40.985 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-21T10:55:41.127 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T10:55:41.142 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T10:55:41.143 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T11:04:32.536 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-01-21T11:04:32.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-01-21T11:04:32.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:04:32.613 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-01-21T11:04:32.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-01-21T11:04:32.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T11:04:32.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:04:32.754 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-01-21T11:04:32.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-01-21T11:04:32.822 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T11:04:32.833 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T11:04:32.834 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T11:04:32.834 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:04:32.836 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T11:04:32.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:04:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T11:04:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T11:04:32.894 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T11:04:32.908 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T11:04:32.949 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-01-21T11:04:32.961 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-01-21T11:04:32.968 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-01-21T11:04:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T11:04:32.980 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-01-21T11:04:32.988 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T11:04:32.994 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T11:04:33.007 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T11:04:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2024-01-21T11:04:33.066 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:04:33.066 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-01-21T11:04:33.066 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.066 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.066 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.066 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.084 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2024-01-21T11:04:33.087 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:04:33.087 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-01-21T11:04:33.088 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.088 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.088 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.088 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:04:33.089 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T11:04:33.098 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T11:04:33.099 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi089.front.sepia.ceph.com 2024-01-21T11:04:33.099 DEBUG:teuthology.orchestra.run.smithi089:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:04:33.099 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2024-01-21T11:04:33.099 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:04:33.859 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:36 ago on Sun 21 Jan 2024 11:01:57 AM UTC. 2024-01-21T11:04:33.885 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:36 ago on Sun 21 Jan 2024 11:01:57 AM UTC. 2024-01-21T11:04:35.800 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:04:35.801 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:04:35.801 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:04:35.801 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:04:35.801 INFO:teuthology.orchestra.run.smithi089.stdout:Resetting modules: 2024-01-21T11:04:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: container-tools 2024-01-21T11:04:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:04:35.802 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:04:35.802 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:04:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:04:35.803 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:04:35.804 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:04:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:04:35.804 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:04:35.804 INFO:teuthology.orchestra.run.smithi178.stdout:Resetting modules: 2024-01-21T11:04:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools 2024-01-21T11:04:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:04:35.805 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:04:35.805 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:04:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:04:36.049 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:04:36.061 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:04:36.870 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:39 ago on Sun 21 Jan 2024 11:01:57 AM UTC. 2024-01-21T11:04:36.874 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:39 ago on Sun 21 Jan 2024 11:01:57 AM UTC. 2024-01-21T11:04:38.794 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:04:38.808 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout:Installing group/module packages: 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T11:04:38.829 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout:Installing module profiles: 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools/common 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2024-01-21T11:04:38.830 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools rhel8 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 27 M 2024-01-21T11:04:38.831 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 89 M 2024-01-21T11:04:38.832 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:04:38.843 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:04:38.843 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:04:38.843 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:04:38.843 INFO:teuthology.orchestra.run.smithi089.stdout:Installing group/module packages: 2024-01-21T11:04:38.843 INFO:teuthology.orchestra.run.smithi089.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T11:04:38.843 INFO:teuthology.orchestra.run.smithi089.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T11:04:38.844 INFO:teuthology.orchestra.run.smithi089.stdout:Installing module profiles: 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout: container-tools/common 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling module streams: 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout: container-tools rhel8 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-01-21T11:04:38.845 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:04:38.846 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 27 M 2024-01-21T11:04:38.846 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 89 M 2024-01-21T11:04:38.846 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:04:42.008 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 253 kB/s | 750 kB 00:02 2024-01-21T11:04:42.032 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 249 kB/s | 750 kB 00:03 2024-01-21T11:04:42.200 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 198 kB/s | 38 kB 00:00 2024-01-21T11:04:42.249 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 175 kB/s | 38 kB 00:00 2024-01-21T11:04:42.950 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 263 kB/s | 1.0 MB 00:03 2024-01-21T11:04:42.975 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 311 kB/s | 241 kB 00:00 2024-01-21T11:04:43.324 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 238 kB/s | 1.0 MB 00:04 2024-01-21T11:04:43.450 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 201 kB/s | 241 kB 00:01 2024-01-21T11:04:43.551 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 309 kB/s | 178 kB 00:00 2024-01-21T11:04:43.742 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 129 kB/s | 25 kB 00:00 2024-01-21T11:04:44.251 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 185 kB/s | 94 kB 00:00 2024-01-21T11:04:44.342 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 199 kB/s | 178 kB 00:00 2024-01-21T11:04:44.525 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 135 kB/s | 25 kB 00:00 2024-01-21T11:04:44.809 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 331 kB/s | 94 kB 00:00 2024-01-21T11:04:58.467 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 265 kB/s | 3.9 MB 00:15 2024-01-21T11:04:59.776 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 238 kB/s | 3.9 MB 00:16 2024-01-21T11:05:07.700 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 327 kB/s | 2.5 MB 00:07 2024-01-21T11:05:07.908 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 253 kB/s | 53 kB 00:00 2024-01-21T11:05:08.058 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 273 kB/s | 41 kB 00:00 2024-01-21T11:05:08.215 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 266 kB/s | 2.5 MB 00:09 2024-01-21T11:05:08.499 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 186 kB/s | 53 kB 00:00 2024-01-21T11:05:08.716 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 189 kB/s | 41 kB 00:00 2024-01-21T11:05:13.107 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 289 kB/s | 9.6 MB 00:34 2024-01-21T11:05:13.333 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 287 kB/s | 9.6 MB 00:34 2024-01-21T11:05:14.816 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 273 kB/s | 8.1 MB 00:30 2024-01-21T11:05:14.817 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:05:14.817 INFO:teuthology.orchestra.run.smithi089.stdout:Total 757 kB/s | 27 MB 00:35 2024-01-21T11:05:15.084 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:05:15.104 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:05:15.104 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:05:15.199 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 274 kB/s | 8.1 MB 00:30 2024-01-21T11:05:15.199 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:05:15.199 INFO:teuthology.orchestra.run.smithi178.stdout:Total 748 kB/s | 27 MB 00:36 2024-01-21T11:05:15.370 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:05:15.373 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:05:15.460 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:05:15.481 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:05:15.481 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:05:15.747 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:05:15.750 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:05:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:05:16.279 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T11:05:16.432 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T11:05:16.434 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:05:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T11:05:16.649 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T11:05:16.853 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T11:05:16.876 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:05:17.039 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T11:05:17.281 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:05:17.395 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T11:05:17.847 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T11:05:18.352 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:05:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T11:05:18.807 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T11:05:18.843 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:05:19.057 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T11:05:19.069 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T11:05:19.274 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T11:05:19.452 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T11:05:19.532 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T11:05:19.945 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T11:05:20.366 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:05:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T11:05:20.747 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:05:20.846 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:05:21.066 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T11:05:21.185 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T11:05:21.186 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:05:21.186 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T11:05:21.244 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:05:21.644 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:05:21.644 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T11:05:21.644 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T11:05:21.644 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T11:05:21.644 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:05:21.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T11:05:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T11:05:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:05:21.707 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:05:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:05:21.708 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:05:21.994 DEBUG:teuthology.parallel:result is None 2024-01-21T11:05:22.179 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T11:05:22.180 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:22.181 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:05:22.181 INFO:teuthology.orchestra.run.smithi178.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T11:05:22.181 INFO:teuthology.orchestra.run.smithi178.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T11:05:22.181 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:05:22.181 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:05:22.444 DEBUG:teuthology.parallel:result is None 2024-01-21T11:05:22.445 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T11:05:22.453 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T11:05:22.454 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi089.front.sepia.ceph.com 2024-01-21T11:05:22.454 DEBUG:teuthology.orchestra.run.smithi089:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:05:22.454 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2024-01-21T11:05:22.454 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:05:22.640 DEBUG:teuthology.parallel:result is None 2024-01-21T11:05:22.687 DEBUG:teuthology.parallel:result is None 2024-01-21T11:05:22.687 INFO:teuthology.run_tasks:Running task install... 2024-01-21T11:05:22.696 DEBUG:teuthology.task.install:project ceph 2024-01-21T11:05:22.696 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T11:05:22.696 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T11:05:22.697 INFO:teuthology.task.install:Using flavor: default 2024-01-21T11:05:22.706 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T11:05:22.706 INFO:teuthology.task.install:extra packages: [] 2024-01-21T11:05:22.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T11:05:22.707 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:05:22.707 INFO:teuthology.packaging:ref: None 2024-01-21T11:05:22.707 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:05:22.707 INFO:teuthology.packaging:branch: None 2024-01-21T11:05:22.707 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:05:22.895 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:05:22.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:05:22.898 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T11:05:22.898 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:05:22.898 INFO:teuthology.packaging:ref: None 2024-01-21T11:05:22.898 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:05:22.898 INFO:teuthology.packaging:branch: None 2024-01-21T11:05:22.899 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:05:22.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:05:23.120 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T11:05:23.120 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T11:05:23.142 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T11:05:23.142 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T11:05:23.185 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T11:05:23.185 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:05:23.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T11:05:23.187 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T11:05:23.188 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:05:23.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T11:05:23.217 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T11:05:23.217 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:05:23.217 INFO:teuthology.packaging:ref: None 2024-01-21T11:05:23.217 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:05:23.217 INFO:teuthology.packaging:branch: None 2024-01-21T11:05:23.217 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:05:23.218 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T11:05:23.218 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T11:05:23.219 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:05:23.219 INFO:teuthology.packaging:ref: None 2024-01-21T11:05:23.219 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:05:23.219 INFO:teuthology.packaging:branch: None 2024-01-21T11:05:23.219 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:05:23.219 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T11:05:23.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T11:05:23.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T11:05:23.363 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T11:05:23.366 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T11:05:23.429 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-01-21T11:05:23.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-01-21T11:05:23.434 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-01-21T11:05:23.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-01-21T11:05:23.700 INFO:teuthology.orchestra.run.smithi178.stdout:76 files removed 2024-01-21T11:05:23.722 INFO:teuthology.orchestra.run.smithi089.stdout:76 files removed 2024-01-21T11:05:23.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2024-01-21T11:05:23.750 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-01-21T11:05:24.208 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-01-21T11:05:24.234 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 937 kB/s | 74 kB 00:00 2024-01-21T11:05:24.300 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 279 kB/s | 15 kB 00:00 2024-01-21T11:05:24.343 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 213 kB/s | 15 kB 00:00 2024-01-21T11:05:24.393 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 27 kB/s | 1.7 kB 00:00 2024-01-21T11:05:24.469 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 18 kB/s | 1.7 kB 00:00 2024-01-21T11:06:46.285 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - AppStream 330 kB/s | 26 MB 01:21 2024-01-21T11:06:52.947 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - AppStream 305 kB/s | 26 MB 01:28 2024-01-21T11:07:29.964 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - BaseOS 269 kB/s | 10 MB 00:39 2024-01-21T11:07:31.839 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras 59 kB/s | 18 kB 00:00 2024-01-21T11:07:32.340 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-21T11:07:33.074 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - PowerTools 7.3 MB/s | 4.7 MB 00:00 2024-01-21T11:07:34.084 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-21T11:07:34.760 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 28 MB/s | 16 MB 00:00 2024-01-21T11:07:38.518 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 517 kB/s | 24 kB 00:00 2024-01-21T11:07:41.373 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - BaseOS 240 kB/s | 10 MB 00:43 2024-01-21T11:07:41.962 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:07:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T11:07:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:07:41.967 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:07:41.968 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:07:41.968 INFO:teuthology.orchestra.run.smithi089.stdout:Install 25 Packages 2024-01-21T11:07:41.968 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-01-21T11:07:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:07:41.968 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 59 M 2024-01-21T11:07:41.968 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:07:42.819 INFO:teuthology.orchestra.run.smithi089.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 42 MB/s | 6.0 MB 00:00 2024-01-21T11:07:42.845 INFO:teuthology.orchestra.run.smithi089.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-21T11:07:42.886 INFO:teuthology.orchestra.run.smithi089.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-21T11:07:42.920 INFO:teuthology.orchestra.run.smithi089.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T11:07:43.029 INFO:teuthology.orchestra.run.smithi089.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2024-01-21T11:07:43.054 INFO:teuthology.orchestra.run.smithi089.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-21T11:07:43.103 INFO:teuthology.orchestra.run.smithi089.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.6 MB/s | 80 kB 00:00 2024-01-21T11:07:43.138 INFO:teuthology.orchestra.run.smithi089.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.0 MB/s | 213 kB 00:00 2024-01-21T11:07:43.172 INFO:teuthology.orchestra.run.smithi089.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T11:07:43.205 INFO:teuthology.orchestra.run.smithi089.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-21T11:07:43.231 INFO:teuthology.orchestra.run.smithi089.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T11:07:43.240 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras 63 kB/s | 18 kB 00:00 2024-01-21T11:07:43.389 INFO:teuthology.orchestra.run.smithi089.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2024-01-21T11:07:43.590 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras common packages 28 kB/s | 6.9 kB 00:00 2024-01-21T11:07:43.657 INFO:teuthology.orchestra.run.smithi089.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-01-21T11:07:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 178 kB/s | 145 kB 00:00 2024-01-21T11:07:44.274 INFO:teuthology.orchestra.run.smithi089.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 195 kB/s | 44 kB 00:00 2024-01-21T11:07:44.517 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - PowerTools 5.7 MB/s | 4.7 MB 00:00 2024-01-21T11:07:44.757 INFO:teuthology.orchestra.run.smithi089.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 236 kB/s | 259 kB 00:01 2024-01-21T11:07:44.874 INFO:teuthology.orchestra.run.smithi089.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 70 kB/s | 42 kB 00:00 2024-01-21T11:07:45.425 INFO:teuthology.orchestra.run.smithi089.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 71 kB/s | 47 kB 00:00 2024-01-21T11:07:45.526 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-21T11:07:45.675 INFO:teuthology.orchestra.run.smithi089.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 100 kB/s | 80 kB 00:00 2024-01-21T11:07:45.800 INFO:teuthology.orchestra.run.smithi089.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 310 kB/s | 116 kB 00:00 2024-01-21T11:07:45.942 INFO:teuthology.orchestra.run.smithi089.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-01-21T11:07:46.017 INFO:teuthology.orchestra.run.smithi089.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 783 kB/s | 59 kB 00:00 2024-01-21T11:07:46.093 INFO:teuthology.orchestra.run.smithi089.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 733 kB/s | 306 kB 00:00 2024-01-21T11:07:46.118 INFO:teuthology.orchestra.run.smithi089.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 750 kB/s | 75 kB 00:00 2024-01-21T11:07:46.218 INFO:teuthology.orchestra.run.smithi089.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-01-21T11:07:46.285 INFO:teuthology.orchestra.run.smithi089.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-01-21T11:07:46.335 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2024-01-21T11:07:46.443 INFO:teuthology.orchestra.run.smithi089.stdout:(27/27): librdkafka-1.6.1-1.el8.x86_64.rpm 222 kB/s | 677 kB 00:03 2024-01-21T11:07:46.447 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:07:46.448 INFO:teuthology.orchestra.run.smithi089.stdout:Total 13 MB/s | 59 MB 00:04 2024-01-21T11:07:46.516 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:07:46.565 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:07:46.565 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:07:46.937 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:07:46.940 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:07:48.071 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:07:48.291 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T11:07:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T11:07:48.667 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:07:49.112 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:07:49.334 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:07:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:07:49.667 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:07:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:07:49.881 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:07:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T11:07:50.135 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 369 kB/s | 24 kB 00:00 2024-01-21T11:07:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T11:07:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:07:50.879 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:07:51.080 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T11:07:51.250 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:07:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:07:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:07:51.675 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:07:52.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:07:52.504 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:07:52.692 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:07:52.793 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:07:52.970 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T11:07:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T11:07:53.349 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:07:53.426 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:07:53.573 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:07:53.575 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T11:07:53.576 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T11:07:53.577 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout:Install 25 Packages 2024-01-21T11:07:53.578 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2024-01-21T11:07:53.579 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:07:53.579 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 59 M 2024-01-21T11:07:53.579 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:07:53.617 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T11:07:53.774 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T11:07:53.970 INFO:teuthology.orchestra.run.smithi178.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2024-01-21T11:07:53.989 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T11:07:53.995 INFO:teuthology.orchestra.run.smithi178.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-21T11:07:54.029 INFO:teuthology.orchestra.run.smithi178.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T11:07:54.062 INFO:teuthology.orchestra.run.smithi178.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T11:07:54.162 INFO:teuthology.orchestra.run.smithi178.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-01-21T11:07:54.188 INFO:teuthology.orchestra.run.smithi178.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-01-21T11:07:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T11:07:54.213 INFO:teuthology.orchestra.run.smithi178.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T11:07:54.238 INFO:teuthology.orchestra.run.smithi178.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-21T11:07:54.284 INFO:teuthology.orchestra.run.smithi178.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 387 kB 00:00 2024-01-21T11:07:54.422 INFO:teuthology.orchestra.run.smithi178.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 21 MB/s | 13 MB 00:00 2024-01-21T11:07:54.447 INFO:teuthology.orchestra.run.smithi178.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 368 kB 00:00 2024-01-21T11:07:54.473 INFO:teuthology.orchestra.run.smithi178.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T11:07:54.732 INFO:teuthology.orchestra.run.smithi178.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-21T11:07:55.715 INFO:teuthology.orchestra.run.smithi178.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 115 kB/s | 145 kB 00:01 2024-01-21T11:07:56.065 INFO:teuthology.orchestra.run.smithi178.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 125 kB/s | 44 kB 00:00 2024-01-21T11:07:56.216 INFO:teuthology.orchestra.run.smithi178.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 175 kB/s | 259 kB 00:01 2024-01-21T11:07:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:07:56.824 INFO:teuthology.orchestra.run.smithi178.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 56 kB/s | 42 kB 00:00 2024-01-21T11:07:56.841 INFO:teuthology.orchestra.run.smithi178.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 75 kB/s | 47 kB 00:00 2024-01-21T11:07:56.925 INFO:teuthology.orchestra.run.smithi178.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 276 kB/s | 677 kB 00:02 2024-01-21T11:07:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:07:57.050 INFO:teuthology.orchestra.run.smithi178.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 557 kB/s | 116 kB 00:00 2024-01-21T11:07:57.108 INFO:teuthology.orchestra.run.smithi178.stdout:(21/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2024-01-21T11:07:57.134 INFO:teuthology.orchestra.run.smithi178.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 2.1 MB/s | 181 kB 00:00 2024-01-21T11:07:57.184 INFO:teuthology.orchestra.run.smithi178.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 781 kB/s | 59 kB 00:00 2024-01-21T11:07:57.209 INFO:teuthology.orchestra.run.smithi178.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 998 kB/s | 75 kB 00:00 2024-01-21T11:07:57.293 INFO:teuthology.orchestra.run.smithi178.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-01-21T11:07:57.368 INFO:teuthology.orchestra.run.smithi178.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-01-21T11:07:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:07:57.710 INFO:teuthology.orchestra.run.smithi178.stdout:(27/27): ndctl-libs-71.1-7.el8.x86_64.rpm 90 kB/s | 80 kB 00:00 2024-01-21T11:07:57.712 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:07:57.713 INFO:teuthology.orchestra.run.smithi178.stdout:Total 14 MB/s | 59 MB 00:04 2024-01-21T11:07:57.781 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:07:57.830 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:07:57.830 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:07:57.982 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:07:58.087 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:07:58.188 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:07:58.191 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:07:58.257 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:07:59.280 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:07:59.511 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T11:07:59.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T11:08:00.082 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:08:00.491 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:08:00.673 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:08:00.777 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:08:01.006 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:08:01.173 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:08:01.239 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:08:01.449 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T11:08:01.982 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T11:08:02.131 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:08:02.211 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:08:02.387 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T11:08:02.565 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:08:02.619 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:08:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:08:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:08:03.181 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:08:03.237 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:08:03.406 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:08:03.822 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:08:04.014 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:08:04.100 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:08:04.284 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T11:08:04.499 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T11:08:04.639 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:08:04.713 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:08:04.939 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T11:08:05.139 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T11:08:05.330 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T11:08:05.539 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T11:08:08.054 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:08:08.256 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:08:08.967 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:08:09.244 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:08:09.349 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:08:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /run 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-01-21T11:08:17.265 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-01-21T11:08:17.266 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2024-01-21T11:08:17.266 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:08:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:08:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:08:18.898 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:08:18.898 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:08:19.579 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:08:19.579 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:08:19.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T11:08:19.581 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.579 INFO:teuthology.orchestra.run.smithi089.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.580 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:20.581 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:08:20.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-01-21T11:08:21.628 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:43 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:08:23.109 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:08:23.109 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:08:23.110 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-01-21T11:08:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:23.111 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 53 M 2024-01-21T11:08:23.111 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 229 M 2024-01-21T11:08:23.111 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:08:23.208 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 169 kB 00:00 2024-01-21T11:08:23.867 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 73 MB/s | 52 MB 00:00 2024-01-21T11:08:24.067 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 219 kB/s | 188 kB 00:00 2024-01-21T11:08:24.092 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 216 kB/s | 203 kB 00:00 2024-01-21T11:08:24.293 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 314 kB/s | 71 kB 00:00 2024-01-21T11:08:25.309 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 224 kB/s | 323 kB 00:01 2024-01-21T11:08:25.310 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:08:25.311 INFO:teuthology.orchestra.run.smithi089.stdout:Total 24 MB/s | 53 MB 00:02 2024-01-21T11:08:25.353 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:08:25.374 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:08:25.374 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:08:25.684 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:08:25.687 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:08:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:08:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T11:08:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T11:08:26.711 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:08:26.773 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:08:26.943 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T11:08:27.095 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /run 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-01-21T11:08:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-01-21T11:08:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2024-01-21T11:08:28.848 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:30.157 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:08:30.216 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:08:30.217 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:08:30.445 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:08:30.445 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:08:31.129 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T11:08:31.130 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T11:08:31.131 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:08:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T11:08:31.132 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T11:08:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:08:32.076 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:08:32.076 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:32.076 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-01-21T11:08:32.076 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.076 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.077 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:32.078 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:08:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:08:32.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2024-01-21T11:08:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:08:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T11:08:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T11:08:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T11:08:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T11:08:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T11:08:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T11:08:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:32.837 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:08:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T11:08:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T11:08:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T11:08:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:32.837 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:08:33.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-01-21T11:08:33.076 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:43 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:08:33.783 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:55 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:08:34.557 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:08:34.559 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-01-21T11:08:34.560 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:34.560 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 53 M 2024-01-21T11:08:34.560 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 229 M 2024-01-21T11:08:34.560 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:08:34.683 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 169 kB 00:00 2024-01-21T11:08:35.277 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:08:35.280 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:08:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:08:35.280 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:08:35.280 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:08:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T11:08:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T11:08:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout:Install 32 Packages 2024-01-21T11:08:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:35.284 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 31 M 2024-01-21T11:08:35.284 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 111 M 2024-01-21T11:08:35.284 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:08:35.300 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 75 MB/s | 52 MB 00:00 2024-01-21T11:08:35.464 INFO:teuthology.orchestra.run.smithi089.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 110 kB/s | 6.3 kB 00:00 2024-01-21T11:08:35.514 INFO:teuthology.orchestra.run.smithi089.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-21T11:08:35.565 INFO:teuthology.orchestra.run.smithi089.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-21T11:08:35.583 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 207 kB/s | 203 kB 00:00 2024-01-21T11:08:35.598 INFO:teuthology.orchestra.run.smithi089.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-01-21T11:08:35.682 INFO:teuthology.orchestra.run.smithi089.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-21T11:08:35.875 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 242 kB/s | 71 kB 00:00 2024-01-21T11:08:35.949 INFO:teuthology.orchestra.run.smithi089.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2024-01-21T11:08:36.050 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 138 kB/s | 188 kB 00:01 2024-01-21T11:08:36.166 INFO:teuthology.orchestra.run.smithi089.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 80 kB/s | 39 kB 00:00 2024-01-21T11:08:36.408 INFO:teuthology.orchestra.run.smithi089.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 194 kB/s | 157 kB 00:00 2024-01-21T11:08:36.584 INFO:teuthology.orchestra.run.smithi089.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 137 kB/s | 57 kB 00:00 2024-01-21T11:08:36.608 INFO:teuthology.orchestra.run.smithi089.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 157 kB/s | 103 kB 00:00 2024-01-21T11:08:36.734 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 225 kB/s | 323 kB 00:01 2024-01-21T11:08:36.735 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:08:36.735 INFO:teuthology.orchestra.run.smithi178.stdout:Total 24 MB/s | 53 MB 00:02 2024-01-21T11:08:36.778 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:08:36.798 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:08:36.798 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:08:37.112 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:08:37.115 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:08:37.613 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:08:37.836 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T11:08:38.027 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T11:08:38.042 INFO:teuthology.orchestra.run.smithi089.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 280 kB/s | 457 kB 00:01 2024-01-21T11:08:38.183 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:08:38.263 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:08:38.457 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T11:08:38.475 INFO:teuthology.orchestra.run.smithi089.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 131 kB/s | 249 kB 00:01 2024-01-21T11:08:38.633 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:08:38.842 INFO:teuthology.orchestra.run.smithi089.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 119 kB/s | 44 kB 00:00 2024-01-21T11:08:39.126 INFO:teuthology.orchestra.run.smithi089.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 653 kB/s | 185 kB 00:00 2024-01-21T11:08:39.260 INFO:teuthology.orchestra.run.smithi089.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 90 kB/s | 109 kB 00:01 2024-01-21T11:08:39.285 INFO:teuthology.orchestra.run.smithi089.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2024-01-21T11:08:39.394 INFO:teuthology.orchestra.run.smithi089.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-01-21T11:08:39.511 INFO:teuthology.orchestra.run.smithi089.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 151 kB/s | 18 kB 00:00 2024-01-21T11:08:39.628 INFO:teuthology.orchestra.run.smithi089.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 417 kB/s | 49 kB 00:00 2024-01-21T11:08:39.745 INFO:teuthology.orchestra.run.smithi089.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 504 kB/s | 59 kB 00:00 2024-01-21T11:08:39.820 INFO:teuthology.orchestra.run.smithi089.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 687 kB/s | 384 kB 00:00 2024-01-21T11:08:39.921 INFO:teuthology.orchestra.run.smithi089.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-01-21T11:08:39.946 INFO:teuthology.orchestra.run.smithi089.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-01-21T11:08:40.021 INFO:teuthology.orchestra.run.smithi089.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-01-21T11:08:40.038 INFO:teuthology.orchestra.run.smithi089.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 260 kB/s | 24 kB 00:00 2024-01-21T11:08:40.122 INFO:teuthology.orchestra.run.smithi089.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-01-21T11:08:40.139 INFO:teuthology.orchestra.run.smithi089.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-01-21T11:08:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2024-01-21T11:08:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2024-01-21T11:08:40.398 INFO:teuthology.orchestra.run.smithi089.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 729 kB/s | 85 kB 00:00 2024-01-21T11:08:40.423 INFO:teuthology.orchestra.run.smithi089.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-01-21T11:08:43.348 INFO:teuthology.orchestra.run.smithi089.stdout:(32/32): python3-cryptography-3.2.1-7.el8.x86_6 95 kB/s | 641 kB 00:06 2024-01-21T11:08:43.349 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:08:43.350 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.8 MB/s | 31 MB 00:08 2024-01-21T11:08:43.430 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:08:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:08:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:08:43.480 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:08:43.706 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:08:43.782 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:08:43.785 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:08:43.999 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:08:43.999 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T11:08:43.999 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T11:08:43.999 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T11:08:43.999 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T11:08:43.999 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:44.472 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:08:44.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2024-01-21T11:08:44.912 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:08:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T11:08:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T11:08:45.453 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:55 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:08:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T11:08:45.764 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T11:08:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T11:08:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T11:08:46.286 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T11:08:46.430 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T11:08:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T11:08:46.860 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T11:08:46.946 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:08:46.949 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:08:46.949 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:08:46.949 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:08:46.949 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:08:46.949 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T11:08:46.950 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T11:08:46.951 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout:Install 32 Packages 2024-01-21T11:08:46.952 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:08:46.953 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 M 2024-01-21T11:08:46.953 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 111 M 2024-01-21T11:08:46.953 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:08:47.047 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T11:08:47.135 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 149 kB/s | 6.3 kB 00:00 2024-01-21T11:08:47.216 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-01-21T11:08:47.261 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.3 MB/s | 1.6 MB 00:00 2024-01-21T11:08:47.270 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T11:08:47.286 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-21T11:08:47.378 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-01-21T11:08:47.604 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 47 MB/s | 18 MB 00:00 2024-01-21T11:08:47.653 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T11:08:47.862 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 273 kB/s | 157 kB 00:00 2024-01-21T11:08:47.887 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 76 kB/s | 39 kB 00:00 2024-01-21T11:08:47.902 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T11:08:48.021 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 248 kB/s | 103 kB 00:00 2024-01-21T11:08:48.080 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 262 kB/s | 57 kB 00:00 2024-01-21T11:08:48.082 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T11:08:48.306 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T11:08:48.475 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:08:48.766 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T11:08:48.955 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T11:08:49.118 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T11:08:49.303 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T11:08:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T11:08:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T11:08:49.763 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 244 kB/s | 457 kB 00:01 2024-01-21T11:08:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T11:08:50.180 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 115 kB/s | 249 kB 00:02 2024-01-21T11:08:50.256 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T11:08:50.413 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 187 kB/s | 44 kB 00:00 2024-01-21T11:08:50.546 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T11:08:50.555 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2024-01-21T11:08:50.622 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.5 MB/s | 173 kB 00:00 2024-01-21T11:08:50.739 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2024-01-21T11:08:50.781 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): python3-pycparser-2.14-14.el8.noarch.r 107 kB/s | 109 kB 00:01 2024-01-21T11:08:50.806 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-01-21T11:08:50.848 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2024-01-21T11:08:50.915 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 881 kB/s | 59 kB 00:00 2024-01-21T11:08:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T11:08:50.982 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.2 MB/s | 283 kB 00:00 2024-01-21T11:08:51.007 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): python3-logutils-0.3.5-11.el8.noarch.r 242 kB/s | 49 kB 00:00 2024-01-21T11:08:51.074 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-01-21T11:08:51.100 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 204 kB/s | 19 kB 00:00 2024-01-21T11:08:51.124 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:08:51.142 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 358 kB/s | 24 kB 00:00 2024-01-21T11:08:51.167 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 436 kB/s | 29 kB 00:00 2024-01-21T11:08:51.209 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 412 kB/s | 27 kB 00:00 2024-01-21T11:08:51.276 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2024-01-21T11:08:51.318 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-01-21T11:08:51.343 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2024-01-21T11:08:51.385 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 344 kB/s | 23 kB 00:00 2024-01-21T11:08:52.652 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:08:52.866 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:08:53.299 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:08:53.482 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:08:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:08:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:08:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:08:54.015 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:08:54.060 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): python3-cryptography-3.2.1-7.el8.x86_6 107 kB/s | 641 kB 00:05 2024-01-21T11:08:54.061 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:08:54.061 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.4 MB/s | 31 MB 00:07 2024-01-21T11:08:54.142 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:08:54.181 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:08:54.181 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T11:08:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T11:08:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T11:08:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T11:08:54.504 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:08:54.505 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:08:55.189 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T11:08:55.190 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T11:08:55.191 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:08:55.192 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:08:55.410 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-01-21T11:08:55.621 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:08:55.826 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T11:08:56.046 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T11:08:56.137 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:18 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:08:56.287 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T11:08:56.469 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T11:08:56.642 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T11:08:56.855 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T11:08:57.024 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T11:08:57.185 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T11:08:57.368 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T11:08:57.584 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:08:57.589 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T11:08:57.630 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:08:57.631 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:08:57.631 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:08:57.805 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T11:08:57.875 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-01-21T11:08:58.051 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T11:08:58.422 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T11:08:58.594 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:20 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:08:58.690 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T11:08:58.895 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T11:08:59.137 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T11:08:59.313 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:08:59.595 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T11:08:59.826 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T11:09:00.006 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T11:09:00.082 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:00.083 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 71 k 2024-01-21T11:09:00.084 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 290 k 2024-01-21T11:09:00.084 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:09:00.124 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-01-21T11:09:00.124 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:09:00.125 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-01-21T11:09:00.125 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:09:00.131 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:09:00.131 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:09:00.133 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:09:00.136 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:09:00.196 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T11:09:00.202 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:09:00.433 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T11:09:00.638 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T11:09:00.691 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:00.864 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T11:09:00.902 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:01.134 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:01.192 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T11:09:01.486 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T11:09:01.514 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:01.514 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:01.514 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:09:01.515 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T11:09:01.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:01.515 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:09:01.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T11:09:01.888 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T11:09:02.092 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:09:02.506 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:09:03.628 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:09:03.817 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:09:04.011 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:09:04.011 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:04.011 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-01-21T11:09:04.011 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:09:04.012 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:04.013 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 163 k 2024-01-21T11:09:04.013 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 587 k 2024-01-21T11:09:04.013 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:09:04.256 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:09:04.784 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:09:04.804 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 206 kB/s | 163 kB 00:00 2024-01-21T11:09:04.805 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:09:04.806 INFO:teuthology.orchestra.run.smithi089.stdout:Total 205 kB/s | 163 kB 00:00 2024-01-21T11:09:04.806 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:09:04.818 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:09:04.818 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:09:04.870 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:09:04.873 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:09:04.994 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:09:05.009 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:09:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:09:05.178 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:09:05.196 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:09:05.319 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:09:05.575 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T11:09:05.727 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T11:09:05.728 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T11:09:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T11:09:05.922 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:09:05.922 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:05.922 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:09:05.923 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:05.923 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:05.923 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:09:06.155 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-01-21T11:09:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T11:09:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:06.483 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:09:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T11:09:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T11:09:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T11:09:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T11:09:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T11:09:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T11:09:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:06.486 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:09:06.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2024-01-21T11:09:06.882 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:28 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:09:07.462 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:17 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:09:08.319 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:09:08.365 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:09:08.366 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:09:08.366 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:09:08.471 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T11:09:08.899 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:09:08.945 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:09:08.946 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:09:08.946 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:09:09.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2024-01-21T11:09:09.190 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:09:09.797 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:19 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:09:10.665 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:09:10.666 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:10.666 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:09:10.667 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-01-21T11:09:10.668 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:10.668 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.0 M 2024-01-21T11:09:10.668 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 15 M 2024-01-21T11:09:10.668 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:09:11.186 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 309 kB/s | 24 kB 00:00 2024-01-21T11:09:11.219 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 80 kB/s | 8.9 kB 00:00 2024-01-21T11:09:11.278 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-01-21T11:09:11.281 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:09:11.282 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:11.282 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:09:11.282 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 71 k 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 290 k 2024-01-21T11:09:11.283 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:09:11.323 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2024-01-21T11:09:11.323 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:09:11.324 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-01-21T11:09:11.324 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:09:11.330 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:09:11.330 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:09:11.332 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:09:11.335 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:09:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:09:11.528 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 109 kB/s | 34 kB 00:00 2024-01-21T11:09:11.620 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 574 kB/s | 196 kB 00:00 2024-01-21T11:09:11.804 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 69 kB/s | 43 kB 00:00 2024-01-21T11:09:11.807 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:09:11.807 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.6 MB/s | 3.0 MB 00:01 2024-01-21T11:09:11.819 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:09:11.830 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:09:11.830 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:09:11.856 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:11.993 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:09:11.997 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:09:12.020 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:12.252 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:09:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T11:09:12.614 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:09:12.615 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:12.615 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:09:12.615 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T11:09:12.615 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:12.615 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:09:12.780 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T11:09:12.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T11:09:12.973 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T11:09:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T11:09:13.570 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:09:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T11:09:14.295 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:09:14.576 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:09:14.576 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:09:14.576 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T11:09:14.576 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T11:09:14.576 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T11:09:14.577 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T11:09:14.952 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T11:09:14.952 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:14.953 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:09:15.053 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:09:15.053 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:15.053 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:09:15.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:15.055 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 163 k 2024-01-21T11:09:15.055 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 587 k 2024-01-21T11:09:15.055 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:09:15.071 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T11:09:15.168 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 1.4 MB/s | 163 kB 00:00 2024-01-21T11:09:15.169 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:09:15.169 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.4 MB/s | 163 kB 00:00 2024-01-21T11:09:15.169 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:09:15.179 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:09:15.179 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:09:15.226 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:09:15.229 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:09:15.788 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:37 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:09:16.047 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:09:16.993 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:09:17.260 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:09:17.261 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-01-21T11:09:17.261 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:09:17.261 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-01-21T11:09:17.261 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:09:17.261 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout:Install 8 Packages 2024-01-21T11:09:17.262 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:09:17.263 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 37 M 2024-01-21T11:09:17.263 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 210 M 2024-01-21T11:09:17.263 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:09:17.452 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:09:17.837 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:09:17.837 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:17.837 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:09:17.837 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T11:09:17.837 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:17.838 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:09:17.893 INFO:teuthology.orchestra.run.smithi089.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 61 MB/s | 8.6 MB 00:00 2024-01-21T11:09:18.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2024-01-21T11:09:18.803 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:28 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:09:20.232 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:09:20.279 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:09:20.280 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:09:20.280 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:09:20.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T11:09:21.123 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:09:22.598 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:09:22.599 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:22.599 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:09:22.600 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:09:22.601 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-01-21T11:09:22.601 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:22.601 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.0 M 2024-01-21T11:09:22.601 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 15 M 2024-01-21T11:09:22.601 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:09:22.869 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 346 kB/s | 24 kB 00:00 2024-01-21T11:09:22.897 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 90 kB/s | 8.9 kB 00:00 2024-01-21T11:09:22.956 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-01-21T11:09:23.014 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 288 kB/s | 34 kB 00:00 2024-01-21T11:09:23.073 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2024-01-21T11:09:23.489 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 68 kB/s | 43 kB 00:00 2024-01-21T11:09:23.491 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:09:23.491 INFO:teuthology.orchestra.run.smithi178.stdout:Total 3.3 MB/s | 3.0 MB 00:00 2024-01-21T11:09:23.502 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:09:23.512 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:09:23.512 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:09:23.692 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:09:23.695 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:09:24.003 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:09:24.205 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T11:09:24.442 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T11:09:24.626 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T11:09:24.755 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T11:09:25.733 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T11:09:25.996 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:09:26.280 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:09:26.280 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:09:26.281 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T11:09:26.281 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T11:09:26.281 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T11:09:26.281 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T11:09:26.665 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:26.666 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:09:26.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T11:09:27.577 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:37 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:09:29.059 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T11:09:29.060 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout:Install 8 Packages 2024-01-21T11:09:29.061 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:09:29.062 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 37 M 2024-01-21T11:09:29.062 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 210 M 2024-01-21T11:09:29.062 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:09:29.602 INFO:teuthology.orchestra.run.smithi178.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 55 MB/s | 8.6 MB 00:00 2024-01-21T11:09:36.336 INFO:teuthology.orchestra.run.smithi089.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 260 kB/s | 4.7 MB 00:18 2024-01-21T11:09:36.761 INFO:teuthology.orchestra.run.smithi089.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 203 kB/s | 3.7 MB 00:18 2024-01-21T11:09:37.245 INFO:teuthology.orchestra.run.smithi089.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 248 kB/s | 225 kB 00:00 2024-01-21T11:09:37.787 INFO:teuthology.orchestra.run.smithi089.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 248 kB/s | 4.9 MB 00:20 2024-01-21T11:09:39.504 INFO:teuthology.orchestra.run.smithi089.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 100 kB/s | 172 kB 00:01 2024-01-21T11:09:41.712 INFO:teuthology.orchestra.run.smithi178.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 316 kB/s | 3.7 MB 00:12 2024-01-21T11:09:42.404 INFO:teuthology.orchestra.run.smithi089.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 125 kB/s | 645 kB 00:05 2024-01-21T11:09:42.454 INFO:teuthology.orchestra.run.smithi178.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 304 kB/s | 225 kB 00:00 2024-01-21T11:09:45.737 INFO:teuthology.orchestra.run.smithi178.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 297 kB/s | 4.7 MB 00:16 2024-01-21T11:09:48.979 INFO:teuthology.orchestra.run.smithi178.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 255 kB/s | 4.9 MB 00:19 2024-01-21T11:09:50.004 INFO:teuthology.orchestra.run.smithi178.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 151 kB/s | 645 kB 00:04 2024-01-21T11:09:51.895 INFO:teuthology.orchestra.run.smithi178.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 59 kB/s | 172 kB 00:02 2024-01-21T11:10:28.594 INFO:teuthology.orchestra.run.smithi089.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 280 kB/s | 14 MB 00:51 2024-01-21T11:10:28.595 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:10:28.595 INFO:teuthology.orchestra.run.smithi089.stdout:Total 532 kB/s | 37 MB 01:11 2024-01-21T11:10:28.821 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:10:28.840 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:10:28.840 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:10:29.168 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:10:29.171 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:10:29.786 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:10:29.934 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:10:30.076 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:10:30.209 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:10:30.809 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:10:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:10:31.633 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:10:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:10:32.549 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:10:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:10:33.133 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:10:34.892 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:10:35.500 INFO:teuthology.orchestra.run.smithi178.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 274 kB/s | 14 MB 00:53 2024-01-21T11:10:35.500 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:10:35.501 INFO:teuthology.orchestra.run.smithi178.stdout:Total 572 kB/s | 37 MB 01:06 2024-01-21T11:10:35.729 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:10:35.748 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:10:35.748 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:10:36.090 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:10:36.093 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:10:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T11:10:36.358 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:10:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:10:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T11:10:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T11:10:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:10:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T11:10:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:10:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T11:10:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T11:10:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:10:36.869 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:10:37.033 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:10:37.067 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T11:10:37.067 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:37.067 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:37.068 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:10:37.176 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-01-21T11:10:37.177 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:10:37.754 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:10:37.906 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:59 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:10:37.960 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:10:38.580 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:10:38.760 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:10:39.379 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T11:10:39.381 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-01-21T11:10:39.382 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:39.383 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 1.9 M 2024-01-21T11:10:39.383 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 24 M 2024-01-21T11:10:39.383 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:10:39.479 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:10:39.805 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:10:39.993 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 524 kB/s | 27 kB 00:00 2024-01-21T11:10:40.052 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:10:40.160 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 122 kB/s | 27 kB 00:00 2024-01-21T11:10:40.186 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 82 kB/s | 20 kB 00:00 2024-01-21T11:10:40.586 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 213 kB/s | 126 kB 00:00 2024-01-21T11:10:40.645 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 227 kB/s | 110 kB 00:00 2024-01-21T11:10:40.945 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 110 kB/s | 33 kB 00:00 2024-01-21T11:10:41.046 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 163 kB/s | 16 kB 00:00 2024-01-21T11:10:41.188 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 574 kB/s | 81 kB 00:00 2024-01-21T11:10:41.446 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 4.5 MB/s | 1.2 MB 00:00 2024-01-21T11:10:41.538 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 701 kB/s | 64 kB 00:00 2024-01-21T11:10:41.614 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 817 kB/s | 61 kB 00:00 2024-01-21T11:10:41.818 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:10:41.989 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 30 kB/s | 43 kB 00:01 2024-01-21T11:10:42.139 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 79 kB/s | 155 kB 00:01 2024-01-21T11:10:42.140 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:10:42.140 INFO:teuthology.orchestra.run.smithi089.stdout:Total 709 kB/s | 1.9 MB 00:02 2024-01-21T11:10:42.197 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:10:42.213 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:10:42.213 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:10:42.301 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:10:42.304 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:10:42.839 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:10:43.042 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T11:10:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T11:10:43.264 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T11:10:43.338 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:10:43.467 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T11:10:43.648 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T11:10:43.649 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T11:10:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T11:10:43.874 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T11:10:44.077 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T11:10:44.077 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:44.078 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:10:44.143 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T11:10:44.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2024-01-21T11:10:44.393 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T11:10:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T11:10:44.923 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:54 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:10:45.653 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T11:10:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T11:10:46.147 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T11:10:46.331 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T11:10:46.405 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:10:46.407 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:10:46.407 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:10:46.407 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:10:46.407 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T11:10:46.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.9 M 2024-01-21T11:10:46.409 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2024-01-21T11:10:46.410 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:10:46.490 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T11:10:46.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T11:10:46.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T11:10:46.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T11:10:46.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T11:10:46.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T11:10:46.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T11:10:46.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T11:10:46.874 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 627 kB/s | 27 kB 00:00 2024-01-21T11:10:47.058 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 88 kB/s | 20 kB 00:00 2024-01-21T11:10:47.074 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 109 kB/s | 27 kB 00:00 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T11:10:47.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:47.231 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:10:47.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T11:10:47.416 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 307 kB/s | 110 kB 00:00 2024-01-21T11:10:47.475 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 210 kB/s | 126 kB 00:00 2024-01-21T11:10:47.659 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 180 kB/s | 33 kB 00:00 2024-01-21T11:10:47.717 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 279 kB/s | 16 kB 00:00 2024-01-21T11:10:47.843 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 650 kB/s | 81 kB 00:00 2024-01-21T11:10:48.001 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 73 kB/s | 43 kB 00:00 2024-01-21T11:10:48.066 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:10 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:10:48.076 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-kubernetes-11.0.0-6.el8.noarch 5.0 MB/s | 1.2 MB 00:00 2024-01-21T11:10:48.135 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-01-21T11:10:48.169 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 384 kB/s | 64 kB 00:00 2024-01-21T11:10:48.685 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 96 kB/s | 155 kB 00:01 2024-01-21T11:10:48.686 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:10:48.686 INFO:teuthology.orchestra.run.smithi178.stdout:Total 859 kB/s | 1.9 MB 00:02 2024-01-21T11:10:48.741 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:10:48.757 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:10:48.757 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:10:48.846 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:10:48.849 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:10:49.328 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:10:49.554 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:10:49.555 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:10:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-01-21T11:10:49.555 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-01-21T11:10:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:10:49.557 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 5.8 M 2024-01-21T11:10:49.557 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 24 M 2024-01-21T11:10:49.557 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:10:49.569 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T11:10:49.783 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-01-21T11:10:49.788 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T11:10:49.873 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 232 kB/s | 21 kB 00:00 2024-01-21T11:10:49.973 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.5 MB/s | 355 kB 00:00 2024-01-21T11:10:50.002 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T11:10:50.007 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.4 MB/s | 49 kB 00:00 2024-01-21T11:10:50.240 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T11:10:50.746 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T11:10:50.965 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T11:10:51.249 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T11:10:51.531 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T11:10:51.565 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 294 kB/s | 538 kB 00:01 2024-01-21T11:10:52.551 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T11:10:52.814 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T11:10:53.023 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T11:10:53.207 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T11:10:53.383 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:10:53.675 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:10:53.675 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T11:10:53.675 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T11:10:53.675 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T11:10:53.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T11:10:53.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T11:10:53.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T11:10:53.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T11:10:53.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T11:10:53.676 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T11:10:53.677 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T11:10:53.677 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T11:10:53.677 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T11:10:54.138 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:54.139 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:10:54.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T11:10:55.033 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:05 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:10:56.490 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T11:10:56.491 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-01-21T11:10:56.492 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:10:56.493 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.8 M 2024-01-21T11:10:56.493 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2024-01-21T11:10:56.493 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:10:56.908 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.5 MB/s | 89 kB 00:00 2024-01-21T11:10:56.975 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 317 kB/s | 21 kB 00:00 2024-01-21T11:10:57.101 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.8 MB/s | 355 kB 00:00 2024-01-21T11:10:57.134 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.4 MB/s | 49 kB 00:00 2024-01-21T11:10:58.543 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 322 kB/s | 538 kB 00:01 2024-01-21T11:11:06.735 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 288 kB/s | 4.8 MB 00:16 2024-01-21T11:11:06.736 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:06.736 INFO:teuthology.orchestra.run.smithi089.stdout:Total 347 kB/s | 5.8 MB 00:17 2024-01-21T11:11:06.790 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:06.800 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:06.800 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:06.903 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:06.906 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:07.462 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T11:11:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T11:11:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T11:11:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T11:11:09.224 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T11:11:09.419 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:11:09.702 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:11:09.702 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:11:09.702 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T11:11:09.702 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T11:11:09.702 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T11:11:09.703 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T11:11:10.087 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:10.088 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:10.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-01-21T11:11:11.511 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:33 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:12.994 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-01-21T11:11:12.995 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T11:11:12.996 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:12.996 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:11:12.996 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:12.996 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-01-21T11:11:12.996 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:12.996 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 968 k 2024-01-21T11:11:12.997 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 3.2 M 2024-01-21T11:11:12.997 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:11:13.245 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-01-21T11:11:13.320 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 298 kB/s | 4.8 MB 00:16 2024-01-21T11:11:13.321 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:13.321 INFO:teuthology.orchestra.run.smithi178.stdout:Total 354 kB/s | 5.8 MB 00:16 2024-01-21T11:11:13.373 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:13.383 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:13.384 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:13.507 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:13.510 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:13.793 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:14.045 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T11:11:14.087 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 93 kB/s | 83 kB 00:00 2024-01-21T11:11:14.087 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:14.087 INFO:teuthology.orchestra.run.smithi089.stdout:Total 887 kB/s | 968 kB 00:01 2024-01-21T11:11:14.098 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:14.119 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:14.119 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:14.207 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:14.210 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:14.297 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T11:11:14.394 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:14.660 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T11:11:14.836 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:11:15.230 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T11:11:15.233 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:11:15.233 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T11:11:15.608 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T11:11:15.617 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T11:11:15.617 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:15.617 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:15.617 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T11:11:15.617 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:15.617 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:15.858 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T11:11:16.036 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:11:16.327 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:11:16.327 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:11:16.327 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T11:11:16.327 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T11:11:16.327 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T11:11:16.327 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T11:11:16.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-01-21T11:11:16.724 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T11:11:16.724 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:16.725 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:16.725 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T11:11:16.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T11:11:16.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T11:11:16.725 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:16.725 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:16.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2024-01-21T11:11:17.107 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:39 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:17.598 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:27 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:18.597 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:18.598 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:11:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:18.599 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 131 k 2024-01-21T11:11:18.599 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 439 k 2024-01-21T11:11:18.599 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:11:18.637 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 131 kB 00:00 2024-01-21T11:11:18.637 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:18.637 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.3 MB/s | 131 kB 00:00 2024-01-21T11:11:18.638 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:18.643 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:18.643 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:18.692 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:18.695 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:18.806 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:19.051 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:19.090 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-01-21T11:11:19.091 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:19.092 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 968 k 2024-01-21T11:11:19.092 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 3.2 M 2024-01-21T11:11:19.092 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:11:19.344 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:19.345 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-01-21T11:11:19.695 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:19.695 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:19.696 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:19.696 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:19.969 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-01-21T11:11:20.137 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 98 kB/s | 83 kB 00:00 2024-01-21T11:11:20.137 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:20.138 INFO:teuthology.orchestra.run.smithi178.stdout:Total 925 kB/s | 968 kB 00:01 2024-01-21T11:11:20.149 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:20.170 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:20.170 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:20.264 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:20.267 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:20.442 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:20.693 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:42 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:20.710 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T11:11:20.902 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:11:21.278 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:11:21.278 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T11:11:21.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T11:11:21.667 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:21.667 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:21.667 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T11:11:21.667 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:21.667 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:21.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2024-01-21T11:11:22.137 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:22.182 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:22.183 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:22.183 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:22.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-01-21T11:11:22.649 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:32 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:23.011 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:45 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:24.136 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:24.137 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:24.137 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 131 k 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 439 k 2024-01-21T11:11:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:11:24.185 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2024-01-21T11:11:24.185 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:24.186 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.7 MB/s | 131 kB 00:00 2024-01-21T11:11:24.186 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:24.191 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:24.191 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:24.241 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:24.244 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:24.354 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:24.502 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:24.502 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:24.502 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:11:24.503 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:24.504 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 24 k 2024-01-21T11:11:24.504 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 94 k 2024-01-21T11:11:24.504 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:11:24.558 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 449 kB/s | 24 kB 00:00 2024-01-21T11:11:24.558 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:24.559 INFO:teuthology.orchestra.run.smithi089.stdout:Total 438 kB/s | 24 kB 00:00 2024-01-21T11:11:24.559 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:24.562 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:24.562 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:24.565 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:24.567 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:24.576 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:24.614 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:24.808 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:24.866 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:25.108 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:25.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:25.248 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:25.248 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:25.248 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:25.248 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:25.248 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:25.450 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:25.450 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:25.450 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:25.450 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:25.450 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:25.450 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:25.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2024-01-21T11:11:25.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-01-21T11:11:26.204 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:36 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:26.289 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:48 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:27.639 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:27.684 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:27.685 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:27.686 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:27.717 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:27.762 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:27.763 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:27.764 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:27.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2024-01-21T11:11:27.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-01-21T11:11:28.518 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:38 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:28.583 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:50 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:29.972 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 24 k 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 94 k 2024-01-21T11:11:29.973 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:11:30.020 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 520 kB/s | 24 kB 00:00 2024-01-21T11:11:30.021 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:30.021 INFO:teuthology.orchestra.run.smithi178.stdout:Total 507 kB/s | 24 kB 00:00 2024-01-21T11:11:30.021 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:30.022 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:30.024 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:30.025 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:30.027 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:30.030 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:30.068 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:30.069 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:30.069 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:30.080 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:30.179 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-01-21T11:11:30.283 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:30.537 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:30.889 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:30.889 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:30.889 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:30.889 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:30.889 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:30.889 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:30.906 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:52 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:31.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2024-01-21T11:11:31.753 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:41 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:32.400 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:32.445 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:32.446 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:32.447 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:32.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-01-21T11:11:33.197 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:33.243 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:33.245 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:33.245 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:33.272 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:55 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:33.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2024-01-21T11:11:34.095 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:44 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:34.705 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:34.751 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:34.752 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:34.752 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:34.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-01-21T11:11:35.532 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:35.578 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:35.579 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:35.579 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:35.610 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:57 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:35.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2024-01-21T11:11:36.444 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:46 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:37.044 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:37.090 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:37.090 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:37.090 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:37.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-01-21T11:11:37.874 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:37.919 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:03:59 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:37.920 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:37.921 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:37.921 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:38.041 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2024-01-21T11:11:38.773 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:48 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:39.356 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:39.402 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:39.403 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-01-21T11:11:39.403 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:39.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-01-21T11:11:40.213 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:04:02 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:40.259 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:40.260 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:40.260 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:40.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2024-01-21T11:11:41.109 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:51 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:41.708 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:41.709 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:11:41.710 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:41.710 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:11:41.710 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:41.710 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 89 k 2024-01-21T11:11:41.710 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 298 k 2024-01-21T11:11:41.710 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:11:41.760 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-01-21T11:11:41.760 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:41.760 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-21T11:11:41.761 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:41.770 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:41.770 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:41.820 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:41.823 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:42.464 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:42.554 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:42.600 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:42.601 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:42.601 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:42.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2024-01-21T11:11:42.787 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:43.180 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:43.180 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:43.181 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:43.181 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:43.181 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:43.181 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:43.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-01-21T11:11:43.459 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:53 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:44.153 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:04:06 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:44.906 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:11:44.951 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:44.952 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-01-21T11:11:44.952 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:45.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2024-01-21T11:11:45.633 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:45.633 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:45.633 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:45.633 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:45.633 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:45.634 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.7 M 2024-01-21T11:11:45.635 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 18 M 2024-01-21T11:11:45.635 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:11:45.724 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2024-01-21T11:11:45.725 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:45.725 INFO:teuthology.orchestra.run.smithi089.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-01-21T11:11:45.725 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:45.737 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:45.737 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:45.791 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:55 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:45.800 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:45.803 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:46.265 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:46.447 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:46.865 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:47.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:47.257 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:47.257 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:47.257 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:47.258 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:47.258 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:47.288 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:47.288 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:47.288 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:47.288 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:47.289 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 89 k 2024-01-21T11:11:47.290 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 298 k 2024-01-21T11:11:47.290 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:11:47.338 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-01-21T11:11:47.339 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:47.339 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-21T11:11:47.339 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:47.349 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:47.349 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:47.399 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:47.402 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:47.505 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:47.574 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-01-21T11:11:47.680 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:47.980 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:48.307 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:04:10 ago on Sun 21 Jan 2024 11:07:38 AM UTC. 2024-01-21T11:11:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:48.411 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:48.411 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:48.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2024-01-21T11:11:49.389 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:59 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:49.790 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-01-21T11:11:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:49.792 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 173 k 2024-01-21T11:11:49.792 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 649 k 2024-01-21T11:11:49.792 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-01-21T11:11:49.847 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 173 kB 00:00 2024-01-21T11:11:49.847 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:49.848 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.0 MB/s | 173 kB 00:00 2024-01-21T11:11:49.848 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-01-21T11:11:49.858 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-01-21T11:11:49.858 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-01-21T11:11:49.908 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-01-21T11:11:49.911 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-01-21T11:11:50.023 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-01-21T11:11:50.229 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:50.517 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:50.878 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:50.878 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:50.878 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:11:50.879 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:50.880 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.7 M 2024-01-21T11:11:50.880 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 18 M 2024-01-21T11:11:50.880 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:11:50.894 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:50.894 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:50.894 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-01-21T11:11:50.894 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:50.894 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-01-21T11:11:50.894 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-01-21T11:11:50.967 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-01-21T11:11:50.968 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:50.968 INFO:teuthology.orchestra.run.smithi178.stdout:Total 42 MB/s | 3.7 MB 00:00 2024-01-21T11:11:50.969 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:50.980 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:50.980 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:51.043 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:51.046 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:51.129 DEBUG:teuthology.parallel:result is None 2024-01-21T11:11:51.512 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:51.714 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:52.122 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:52.534 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:52.534 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:52.534 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:52.534 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:52.534 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:52.534 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:52.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2024-01-21T11:11:53.523 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:03 ago on Sun 21 Jan 2024 11:07:50 AM UTC. 2024-01-21T11:11:54.983 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-01-21T11:11:54.983 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-01-21T11:11:54.983 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:54.983 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-01-21T11:11:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T11:11:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-01-21T11:11:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-01-21T11:11:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-01-21T11:11:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 173 k 2024-01-21T11:11:54.985 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 649 k 2024-01-21T11:11:54.985 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-01-21T11:11:55.146 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2024-01-21T11:11:55.146 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:11:55.147 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.0 MB/s | 173 kB 00:00 2024-01-21T11:11:55.147 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-01-21T11:11:55.157 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-01-21T11:11:55.157 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-01-21T11:11:55.206 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-01-21T11:11:55.209 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-01-21T11:11:55.321 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-01-21T11:11:55.520 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:55.811 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:56.178 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:11:56.178 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:56.178 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-01-21T11:11:56.178 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:11:56.178 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:11:56.178 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-01-21T11:11:56.419 DEBUG:teuthology.parallel:result is None 2024-01-21T11:11:56.419 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:11:56.419 INFO:teuthology.packaging:ref: None 2024-01-21T11:11:56.420 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:11:56.420 INFO:teuthology.packaging:branch: None 2024-01-21T11:11:56.420 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:11:56.420 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:11:56.755 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T11:11:56.782 INFO:teuthology.orchestra.run.smithi089.stdout:16.2.4-0.el8 2024-01-21T11:11:56.782 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T11:11:56.782 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T11:11:56.784 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:11:56.784 INFO:teuthology.packaging:ref: None 2024-01-21T11:11:56.784 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:11:56.784 INFO:teuthology.packaging:branch: None 2024-01-21T11:11:56.784 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:11:56.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:11:56.920 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T11:11:56.944 INFO:teuthology.orchestra.run.smithi178.stdout:16.2.4-0.el8 2024-01-21T11:11:56.944 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T11:11:56.944 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T11:11:56.946 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T11:11:56.946 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:11:56.946 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T11:11:56.979 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:11:56.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T11:11:57.012 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T11:11:57.012 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:11:57.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T11:11:57.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T11:11:57.118 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:11:57.118 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T11:11:57.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T11:11:57.212 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T11:11:57.213 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:11:57.213 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T11:11:57.241 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T11:11:57.309 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:11:57.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T11:11:57.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T11:11:57.403 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T11:11:57.404 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:11:57.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T11:11:57.432 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T11:11:57.499 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:11:57.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T11:11:57.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T11:11:57.593 INFO:teuthology.run_tasks:Running task print... 2024-01-21T11:11:57.604 INFO:teuthology.task.print:**** done install task... 2024-01-21T11:11:57.605 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T11:11:57.730 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T11:11:57.731 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T11:11:57.731 INFO:tasks.cephadm:Cluster fsid is e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:11:57.731 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T11:11:57.731 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T11:11:57.732 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi089': '172.21.15.89', 'mon.smithi178': '172.21.15.178'} 2024-01-21T11:11:57.732 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T11:11:57.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2024-01-21T11:11:57.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2024-01-21T11:11:57.788 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T11:11:57.788 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T11:11:58.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 11:11 /home/ubuntu/cephtest/cephadm 2024-01-21T11:11:58.172 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T11:11:58.415 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 11:11 /home/ubuntu/cephtest/cephadm 2024-01-21T11:11:58.416 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T11:11:58.436 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T11:11:58.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2024-01-21T11:11:58.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2024-01-21T11:11:58.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2024-01-21T11:11:58.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2024-01-21T11:11:58.600 INFO:tasks.cephadm:Writing seed config... 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T11:11:58.602 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T11:11:58.603 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T11:11:58.604 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T11:11:58.605 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T11:11:58.606 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:11:58.606 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T11:11:58.631 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e44d1d4c-b84d-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T11:11:58.632 DEBUG:teuthology.orchestra.run.smithi089:mon.smithi089> sudo journalctl -f -n 0 -u ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089.service 2024-01-21T11:11:58.674 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T11:11:58.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.89 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T11:11:58.700 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:-- Logs begin at Sun 2024-01-21 10:54:32 UTC. -- 2024-01-21T11:11:58.955 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T11:11:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: 4.6.1 2024-01-21T11:11:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:Verifying podman|docker is present... 2024-01-21T11:11:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T11:11:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: 4.6.1 2024-01-21T11:11:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:Verifying lvm2 is present... 2024-01-21T11:11:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:Verifying time synchronization is in place... 2024-01-21T11:11:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T11:11:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T11:11:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active chrony.service 2024-01-21T11:11:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: inactive 2024-01-21T11:11:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T11:11:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: enabled 2024-01-21T11:11:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T11:11:59.962 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: active 2024-01-21T11:11:59.963 INFO:teuthology.orchestra.run.smithi089.stderr:Unit chronyd.service is enabled and running 2024-01-21T11:11:59.963 INFO:teuthology.orchestra.run.smithi089.stderr:Repeating the final host check... 2024-01-21T11:11:59.963 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T11:11:59.983 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: 4.6.1 2024-01-21T11:11:59.998 INFO:teuthology.orchestra.run.smithi089.stderr:podman|docker (/bin/podman) is present 2024-01-21T11:11:59.998 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl is present 2024-01-21T11:11:59.998 INFO:teuthology.orchestra.run.smithi089.stderr:lvcreate is present 2024-01-21T11:11:59.998 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T11:12:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T11:12:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active chrony.service 2024-01-21T11:12:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: inactive 2024-01-21T11:12:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T11:12:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: enabled 2024-01-21T11:12:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T11:12:00.021 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: active 2024-01-21T11:12:00.022 INFO:teuthology.orchestra.run.smithi089.stderr:Unit chronyd.service is enabled and running 2024-01-21T11:12:00.022 INFO:teuthology.orchestra.run.smithi089.stderr:Host looks OK 2024-01-21T11:12:00.022 INFO:teuthology.orchestra.run.smithi089.stderr:Cluster fsid: e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:00.023 INFO:teuthology.orchestra.run.smithi089.stderr:Acquiring lock 140106901137896 on /run/cephadm/e44d1d4c-b84d-11ee-95b0-87774f69a715.lock 2024-01-21T11:12:00.023 INFO:teuthology.orchestra.run.smithi089.stderr:Lock 140106901137896 acquired on /run/cephadm/e44d1d4c-b84d-11ee-95b0-87774f69a715.lock 2024-01-21T11:12:00.023 INFO:teuthology.orchestra.run.smithi089.stderr:Verifying IP 172.21.15.89 port 3300 ... 2024-01-21T11:12:00.024 INFO:teuthology.orchestra.run.smithi089.stderr:Verifying IP 172.21.15.89 port 6789 ... 2024-01-21T11:12:00.024 INFO:teuthology.orchestra.run.smithi089.stderr:Base mon IP is 172.21.15.89, final addrv is [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-01-21T11:12:00.024 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /sbin/ip route ls 2024-01-21T11:12:00.026 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.89 metric 100 2024-01-21T11:12:00.026 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.89 metric 100 2024-01-21T11:12:00.027 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T11:12:00.029 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T11:12:00.029 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-21T11:12:00.029 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-01-21T11:12:00.029 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T11:12:00.032 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T11:12:00.032 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T11:12:00.032 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T11:12:00.032 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-21T11:12:00.032 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-01-21T11:12:00.032 INFO:teuthology.orchestra.run.smithi089.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T11:12:00.033 INFO:teuthology.orchestra.run.smithi089.stderr:Mon IP 172.21.15.89 is in CIDR network 172.21.0.0/20 2024-01-21T11:12:00.033 INFO:teuthology.orchestra.run.smithi089.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T11:12:00.034 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T11:12:00.034 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T11:12:00.120 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T11:12:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: Getting image source signatures 2024-01-21T11:12:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T11:12:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T11:12:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T11:12:20.641 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T11:12:21.009 INFO:teuthology.orchestra.run.smithi089.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T11:12:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T11:12:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T11:12:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T11:12:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T11:12:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T11:12:22.131 INFO:teuthology.orchestra.run.smithi089.stderr:stat: 167 167 2024-01-21T11:12:22.440 INFO:teuthology.orchestra.run.smithi089.stderr:Creating initial keys... 2024-01-21T11:12:22.440 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T11:12:22.714 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-authtool: AQAW/KxlNWuZKRAA4o1UmIVmU8ggacGwubfAjA== 2024-01-21T11:12:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T11:12:23.331 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-authtool: AQAX/Kxl577kEhAAZ1w+HT3rNv+C6B3ew4VhTw== 2024-01-21T11:12:23.658 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T11:12:23.955 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-authtool: AQAX/Kxl7UT2NxAAbpEEKgmJDJcDS6j5x5QD5A== 2024-01-21T11:12:24.285 INFO:teuthology.orchestra.run.smithi089.stderr:Creating initial monmap... 2024-01-21T11:12:24.285 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpxgyrwzrf:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 --addv smithi089 [v2:172.21.15.89:3300,v1:172.21.15.89:6789] /tmp/monmap 2024-01-21T11:12:24.539 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T11:12:24.540 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:24.540 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T11:12:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:monmaptool for smithi089 [v2:172.21.15.89:3300,v1:172.21.15.89:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T11:12:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/monmaptool: set fsid to e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T11:12:24.884 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:12:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:Creating mon... 2024-01-21T11:12:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/mon.smithi089:/var/lib/ceph/mon/ceph-smithi089:z -v /tmp/ceph-tmpgwq6_2x0:/tmp/keyring:z -v /tmp/ceph-tmpxgyrwzrf:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi089 --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T11:12:25.180 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.150+0000 7f5bc4606700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T11:12:25.210 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.150+0000 7f5bc4606700 1 imported monmap: 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T11:12:24.527360+0000 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: created 2024-01-21T11:12:24.527360+0000 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.smithi089 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.150+0000 7f5bc4606700 0 /usr/bin/ceph-mon: set fsid to e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Compile date May 13 2021 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: DB SUMMARY 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi089/store.db dir, Total Num: 0, files: 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi089/store.db: 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.env: 0x5561dee151c0 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.fs: Posix File System 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.info_log: 0x5561dfb9eec0 2024-01-21T11:12:25.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.statistics: (nil) 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.use_fsync: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T11:12:25.213 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.db_log_dir: 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi089/store.db 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.write_buffer_manager: 0x5561dfba5530 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T11:12:25.214 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.unordered_write: 0 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.row_cache: None 2024-01-21T11:12:25.215 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.wal_filter: None 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T11:12:25.216 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.max_open_files: -1 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Compression algorithms supported: 2024-01-21T11:12:25.217 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kZSTD supported: 0 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.151+0000 7f5bc4606700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi089/store.db/MANIFEST-000001 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.218 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.merge_operator: 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_filter: None 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5561dfab6120) 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T11:12:25.219 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_cache: 0x5561dfaecd10 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T11:12:25.220 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T11:12:25.221 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression: NoCompression 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.num_levels: 7 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:12:25.222 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T11:12:25.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T11:12:25.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T11:12:25.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.ttl: 2592000 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi089/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.152+0000 7f5bc4606700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.153+0000 7f5bc4606700 4 rocksdb: DB pointer 0x5561dfbb1800 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.153+0000 7f5bad49f700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T11:12:25.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.153+0000 7f5bad49f700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:12:25.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:12:25.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:25.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.154+0000 7f5bc4606700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.154+0000 7f5bc4606700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T11:12:25.230 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:12:25.154+0000 7f5bc4606700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi089 for mon.smithi089 2024-01-21T11:12:25.523 INFO:teuthology.orchestra.run.smithi089.stderr:create mon.smithi089 on 2024-01-21T11:12:25.523 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:25.528 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl disable ceph.target 2024-01-21T11:12:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T11:12:25.621 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl enable ceph.target 2024-01-21T11:12:25.629 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T11:12:25.707 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl start ceph.target 2024-01-21T11:12:25.714 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl enable ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target 2024-01-21T11:12:25.720 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target. 2024-01-21T11:12:25.720 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target. 2024-01-21T11:12:25.798 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target 2024-01-21T11:12:25.805 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl daemon-reload 2024-01-21T11:12:25.891 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089 2024-01-21T11:12:25.914 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl reset-failed ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089 2024-01-21T11:12:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Failed to reset failed state of unit ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089.service: Unit ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089.service not loaded. 2024-01-21T11:12:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl enable ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089 2024-01-21T11:12:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target.wants/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089.service → /etc/systemd/system/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@.service. 2024-01-21T11:12:26.015 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089 2024-01-21T11:12:26.326 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:12:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: disabled 2024-01-21T11:12:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:12:26.342 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: inactive 2024-01-21T11:12:26.342 INFO:teuthology.orchestra.run.smithi089.stderr:firewalld.service is not enabled 2024-01-21T11:12:26.343 INFO:teuthology.orchestra.run.smithi089.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T11:12:26.343 INFO:teuthology.orchestra.run.smithi089.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T11:12:26.343 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for mon to start... 2024-01-21T11:12:26.343 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for mon... 2024-01-21T11:12:26.343 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/mon.smithi089:/var/lib/ceph/mon/ceph-smithi089:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T11:12:26.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: cluster 2024-01-21T11:12:26.342125+0000 mon.smithi089 (mon.0) 1 : cluster [INF] mon.smithi089 is new leader, mons smithi089 in quorum (ranks 0) 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.738+0000 7f6df53f5700 1 Processor -- start 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.738+0000 7f6df53f5700 1 -- start start 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.738+0000 7f6df53f5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.738+0000 7f6df53f5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00fed30 con 0x7f6df00703f0 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6deeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6deeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fe840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58712/0 (socket says 172.21.15.89:58712) 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6deeffd700 1 -- 172.21.15.89:0/4100245132 learned_addr learned my addr 172.21.15.89:0/4100245132 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:26.742 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6deeffd700 1 -- 172.21.15.89:0/4100245132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00fee70 con 0x7f6df00703f0 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6deeffd700 1 --2- 172.21.15.89:0/4100245132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fe840 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6de400b5f0 tx=0x7f6de4006ee0).ready entity=mon.0 client_cookie=a78b98465d488f48 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6dedffb700 1 -- 172.21.15.89:0/4100245132 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de400d600 con 0x7f6df00703f0 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6dedffb700 1 -- 172.21.15.89:0/4100245132 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de400d760 con 0x7f6df00703f0 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.739+0000 7f6dedffb700 1 -- 172.21.15.89:0/4100245132 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de400da30 con 0x7f6df00703f0 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 -- 172.21.15.89:0/4100245132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 msgr2=0x7f6df00fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 --2- 172.21.15.89:0/4100245132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fe840 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6de400b5f0 tx=0x7f6de4006ee0).stop 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 -- 172.21.15.89:0/4100245132 shutdown_connections 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 --2- 172.21.15.89:0/4100245132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fe840 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 -- 172.21.15.89:0/4100245132 >> 172.21.15.89:0/4100245132 conn(0x7f6df00f62a0 msgr2=0x7f6df00f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 -- 172.21.15.89:0/4100245132 shutdown_connections 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.740+0000 7f6df53f5700 1 -- 172.21.15.89:0/4100245132 wait complete. 2024-01-21T11:12:26.743 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6df53f5700 1 Processor -- start 2024-01-21T11:12:26.744 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6df53f5700 1 -- start start 2024-01-21T11:12:26.744 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6df53f5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:26.744 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6df53f5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00fa740 con 0x7f6df00703f0 2024-01-21T11:12:26.744 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6deeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:26.744 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6deeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58722/0 (socket says 172.21.15.89:58722) 2024-01-21T11:12:26.744 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6deeffd700 1 -- 172.21.15.89:0/3767200198 learned_addr learned my addr 172.21.15.89:0/3767200198 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6deeffd700 1 -- 172.21.15.89:0/3767200198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de400b200 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6deeffd700 1 --2- 172.21.15.89:0/3767200198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fc2c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6de4006c70 tx=0x7f6de4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.741+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de4011c80 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de4011de0 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df00fa8e0 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de401d5d0 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df00fad70 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6de400f070 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6de4027460 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.742+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0059100 con 0x7f6df00703f0 2024-01-21T11:12:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.745+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6de4016070 con 0x7f6df00703f0 2024-01-21T11:12:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.811+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6df0027a00 con 0x7f6df00703f0 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.811+0000 7f6dd77fe700 1 -- 172.21.15.89:0/3767200198 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f6de401d730 con 0x7f6df00703f0 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: cluster: 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: id: e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: services: 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi089 (age 0.465291s) 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: data: 2024-01-21T11:12:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: pgs: 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 msgr2=0x7f6df00fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 --2- 172.21.15.89:0/3767200198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fc2c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6de4006c70 tx=0x7f6de4016000).stop 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 shutdown_connections 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 --2- 172.21.15.89:0/3767200198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df00703f0 0x7f6df00fc2c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 >> 172.21.15.89:0/3767200198 conn(0x7f6df00f62a0 msgr2=0x7f6df00f7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 shutdown_connections 2024-01-21T11:12:26.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:26.812+0000 7f6df53f5700 1 -- 172.21.15.89:0/3767200198 wait complete. 2024-01-21T11:12:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:mon is available 2024-01-21T11:12:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T11:12:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z -v /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/mon.smithi089:/var/lib/ceph/mon/ceph-smithi089:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi089/config 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.577+0000 7f57467cb700 1 Processor -- start 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.577+0000 7f57467cb700 1 -- start start 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.577+0000 7f57467cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f57400fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.577+0000 7f57467cb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57400fb490 con 0x7f57400fab80 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f57400fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f57400fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58732/0 (socket says 172.21.15.89:58732) 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573ffff700 1 -- 172.21.15.89:0/2031168357 learned_addr learned my addr 172.21.15.89:0/2031168357 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:27.579 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573ffff700 1 -- 172.21.15.89:0/2031168357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400fb5d0 con 0x7f57400fab80 2024-01-21T11:12:27.580 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573ffff700 1 --2- 172.21.15.89:0/2031168357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f57400fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f573400b5f0 tx=0x7f5734006ee0).ready entity=mon.0 client_cookie=3bf0b34cff54fc0a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:27.580 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573effd700 1 -- 172.21.15.89:0/2031168357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f573400d600 con 0x7f57400fab80 2024-01-21T11:12:27.581 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573effd700 1 -- 172.21.15.89:0/2031168357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573400d760 con 0x7f57400fab80 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.578+0000 7f573effd700 1 -- 172.21.15.89:0/2031168357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f573400da30 con 0x7f57400fab80 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 -- 172.21.15.89:0/2031168357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 msgr2=0x7f57400fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 --2- 172.21.15.89:0/2031168357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f57400fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f573400b5f0 tx=0x7f5734006ee0).stop 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 -- 172.21.15.89:0/2031168357 shutdown_connections 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 --2- 172.21.15.89:0/2031168357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f57400fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 -- 172.21.15.89:0/2031168357 >> 172.21.15.89:0/2031168357 conn(0x7f57400f63c0 msgr2=0x7f57400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 -- 172.21.15.89:0/2031168357 shutdown_connections 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 -- 172.21.15.89:0/2031168357 wait complete. 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.579+0000 7f57467cb700 1 Processor -- start 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f57467cb700 1 -- start start 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f57467cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f574010cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f57467cb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57400704b0 con 0x7f57400fab80 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f573ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f574010cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:27.582 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f573ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f574010cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58738/0 (socket says 172.21.15.89:58738) 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f573ffff700 1 -- 172.21.15.89:0/3246313944 learned_addr learned my addr 172.21.15.89:0/3246313944 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f573ffff700 1 -- 172.21.15.89:0/3246313944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573400b200 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.580+0000 7f573ffff700 1 --2- 172.21.15.89:0/3246313944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f574010cf10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f5734006c70 tx=0x7f5734016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5734011b40 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734011ca0 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57400706b0 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f573401d430 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574010d5c0 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f573400f070 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.581+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5734027460 con 0x7f57400fab80 2024-01-21T11:12:27.583 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.582+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5740045ad0 con 0x7f57400fab80 2024-01-21T11:12:27.584 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.584+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5734016070 con 0x7f57400fab80 2024-01-21T11:12:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.648+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f57400fbec0 con 0x7f57400fab80 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.651+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f573401d590 con 0x7f57400fab80 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.651+0000 7f573cff9700 1 -- 172.21.15.89:0/3246313944 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f57340276f0 con 0x7f57400fab80 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: [global] 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: fsid = e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: [mgr] 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: [osd] 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 msgr2=0x7f574010cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 --2- 172.21.15.89:0/3246313944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f574010cf10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f5734006c70 tx=0x7f5734016000).stop 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 shutdown_connections 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 --2- 172.21.15.89:0/3246313944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57400fab80 0x7f574010cf10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 >> 172.21.15.89:0/3246313944 conn(0x7f57400f63c0 msgr2=0x7f57400fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 shutdown_connections 2024-01-21T11:12:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:27.652+0000 7f57467cb700 1 -- 172.21.15.89:0/3246313944 wait complete. 2024-01-21T11:12:27.671 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: cluster 2024-01-21T11:12:26.346494+0000 mon.smithi089 (mon.0) 2 : cluster [INF] mon.smithi089 is new leader, mons smithi089 in quorum (ranks 0) 2024-01-21T11:12:27.671 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: cluster 2024-01-21T11:12:26.346909+0000 mon.smithi089 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} 2024-01-21T11:12:27.671 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: cluster 2024-01-21T11:12:26.350235+0000 mon.smithi089 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T11:12:27.671 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: cluster 2024-01-21T11:12:26.352364+0000 mon.smithi089 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T11:12:27.671 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: cluster 2024-01-21T11:12:26.352825+0000 mon.smithi089 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T11:12:27.671 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: audit 2024-01-21T11:12:26.811741+0000 mon.smithi089 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.89:0/3767200198' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T11:12:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:Generating new minimal ceph.conf... 2024-01-21T11:12:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z -v /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/mon.smithi089:/var/lib/ceph/mon/ceph-smithi089:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi089/config 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52d5ac9700 1 Processor -- start 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52d5ac9700 1 -- start start 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52d5ac9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d00fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52d5ac9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0070790 con 0x7f52d0070370 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d00fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d00fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58744/0 (socket says 172.21.15.89:58744) 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52cf7fe700 1 -- 172.21.15.89:0/2638012480 learned_addr learned my addr 172.21.15.89:0/2638012480 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.343+0000 7f52cf7fe700 1 -- 172.21.15.89:0/2638012480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00fee90 con 0x7f52d0070370 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52cf7fe700 1 --2- 172.21.15.89:0/2638012480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d00fe9a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f52c000b5f0 tx=0x7f52c0006ee0).ready entity=mon.0 client_cookie=3441fac15e643956 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52ce7fc700 1 -- 172.21.15.89:0/2638012480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c000d600 con 0x7f52d0070370 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52ce7fc700 1 -- 172.21.15.89:0/2638012480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f52c000dbd0 con 0x7f52d0070370 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52ce7fc700 1 -- 172.21.15.89:0/2638012480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c0011c00 con 0x7f52d0070370 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2638012480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 msgr2=0x7f52d00fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52d5ac9700 1 --2- 172.21.15.89:0/2638012480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d00fe9a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f52c000b5f0 tx=0x7f52c0006ee0).stop 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2638012480 shutdown_connections 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52d5ac9700 1 --2- 172.21.15.89:0/2638012480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d00fe9a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.344+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2638012480 >> 172.21.15.89:0/2638012480 conn(0x7f52d00f63c0 msgr2=0x7f52d00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.345+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2638012480 shutdown_connections 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.345+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2638012480 wait complete. 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.345+0000 7f52d5ac9700 1 Processor -- start 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52d5ac9700 1 -- start start 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52d5ac9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d010cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52d5ac9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d010d070 con 0x7f52d0070370 2024-01-21T11:12:28.349 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d010cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d010cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58750/0 (socket says 172.21.15.89:58750) 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52cf7fe700 1 -- 172.21.15.89:0/2215820631 learned_addr learned my addr 172.21.15.89:0/2215820631 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52cf7fe700 1 -- 172.21.15.89:0/2215820631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c000b200 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52cf7fe700 1 --2- 172.21.15.89:0/2215820631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d010cb80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52c000b190 tx=0x7f52c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c001b410 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d010d270 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f52c001b9e0 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c001a8f0 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.346+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d010d680 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.347+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f52c000f070 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.347+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f52c0025cf0 con 0x7f52d0070370 2024-01-21T11:12:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.348+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0045ad0 con 0x7f52d0070370 2024-01-21T11:12:28.351 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.350+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f52c0016070 con 0x7f52d0070370 2024-01-21T11:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.414+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f52d0106c30 con 0x7f52d0070370 2024-01-21T11:12:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.414+0000 7f52b7fff700 1 -- 172.21.15.89:0/2215820631 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f52c0025460 con 0x7f52d0070370 2024-01-21T11:12:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 msgr2=0x7f52d010cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 --2- 172.21.15.89:0/2215820631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d010cb80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52c000b190 tx=0x7f52c0006ee0).stop 2024-01-21T11:12:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 shutdown_connections 2024-01-21T11:12:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 --2- 172.21.15.89:0/2215820631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52d0070370 0x7f52d010cb80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 >> 172.21.15.89:0/2215820631 conn(0x7f52d00f63c0 msgr2=0x7f52d00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 shutdown_connections 2024-01-21T11:12:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:28.415+0000 7f52d5ac9700 1 -- 172.21.15.89:0/2215820631 wait complete. 2024-01-21T11:12:28.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: audit 2024-01-21T11:12:27.648916+0000 mon.smithi089 (mon.0) 8 : audit [INF] from='client.? 172.21.15.89:0/3246313944' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T11:12:28.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: audit 2024-01-21T11:12:27.651910+0000 mon.smithi089 (mon.0) 9 : audit [INF] from='client.? 172.21.15.89:0/3246313944' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T11:12:28.783 INFO:teuthology.orchestra.run.smithi089.stderr:Restarting the monitor... 2024-01-21T11:12:28.783 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl restart ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089 2024-01-21T11:12:29.181 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:28 smithi089 systemd[1]: Stopping Ceph mon.smithi089 for e44d1d4c-b84d-11ee-95b0-87774f69a715... 2024-01-21T11:12:29.181 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: debug 2024-01-21T11:12:28.863+0000 7f29952d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi089 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T11:12:29.181 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38436]: debug 2024-01-21T11:12:28.863+0000 7f29952d0700 -1 mon.smithi089@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T11:12:29.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 podman[38756]: ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089 2024-01-21T11:12:29.434 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 systemd[1]: ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089.service: Succeeded. 2024-01-21T11:12:29.434 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 systemd[1]: Stopped Ceph mon.smithi089 for e44d1d4c-b84d-11ee-95b0-87774f69a715. 2024-01-21T11:12:29.434 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 systemd[1]: Starting Ceph mon.smithi089 for e44d1d4c-b84d-11ee-95b0-87774f69a715... 2024-01-21T11:12:29.633 INFO:teuthology.orchestra.run.smithi089.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T11:12:29.634 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T11:12:29.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.623+0000 7faa87977700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T11:12:29.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.623+0000 7faa87977700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T11:12:29.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.623+0000 7faa87977700 0 pidfile_write: ignore empty --pid-file 2024-01-21T11:12:29.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.627+0000 7faa87977700 0 load: jerasure load: lrc load: isa 2024-01-21T11:12:29.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T11:12:29.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T11:12:29.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Compile date May 13 2021 2024-01-21T11:12:29.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: DB SUMMARY 2024-01-21T11:12:29.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T11:12:29.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T11:12:29.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.941 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T11:12:29.941 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.941 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi089/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T11:12:29.941 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.941 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi089/store.db: 000006.log size: 92918 ; 2024-01-21T11:12:29.941 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.env: 0x56028dfb31c0 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.fs: Posix File System 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.info_log: 0x560290231960 2024-01-21T11:12:29.942 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.statistics: (nil) 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.use_fsync: 0 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T11:12:29.943 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T11:12:29.944 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T11:12:29.944 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T11:12:29.944 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T11:12:29.944 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T11:12:29.944 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T11:12:29.944 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T11:12:29.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.db_log_dir: 2024-01-21T11:12:29.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi089/store.db 2024-01-21T11:12:29.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T11:12:29.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T11:12:29.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T11:12:29.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T11:12:29.946 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T11:12:29.946 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T11:12:29.946 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T11:12:29.946 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T11:12:29.946 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T11:12:29.946 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T11:12:29.947 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.write_buffer_manager: 0x560290ea6030 2024-01-21T11:12:29.947 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T11:12:29.947 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T11:12:29.947 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T11:12:29.947 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 bash[38827]: c04570de3f1cdc1291402eabc3b17ddc6c9c15ef94d6f02c57df8150da429549 2024-01-21T11:12:29.947 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T11:12:29.948 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T11:12:29.948 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T11:12:29.948 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T11:12:29.948 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T11:12:29.948 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T11:12:29.948 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.unordered_write: 0 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.row_cache: None 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.wal_filter: None 2024-01-21T11:12:29.949 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T11:12:29.950 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T11:12:29.951 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T11:12:29.952 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T11:12:29.952 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T11:12:29.952 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T11:12:29.952 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T11:12:29.952 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_open_files: -1 2024-01-21T11:12:29.952 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Compression algorithms supported: 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kZSTD supported: 0 2024-01-21T11:12:29.953 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T11:12:29.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T11:12:29.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T11:12:29.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T11:12:29.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T11:12:29.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T11:12:29.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T11:12:29.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi089/store.db/MANIFEST-000005 2024-01-21T11:12:29.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T11:12:29.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T11:12:29.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.merge_operator: 2024-01-21T11:12:29.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_filter: None 2024-01-21T11:12:29.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T11:12:29.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T11:12:29.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T11:12:29.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560290148128) 2024-01-21T11:12:29.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cache_index_and_filter_blocks: 1 2024-01-21T11:12:29.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T11:12:29.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T11:12:29.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: pin_top_level_index_and_filter: 1 2024-01-21T11:12:29.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: index_type: 0 2024-01-21T11:12:29.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: data_block_index_type: 0 2024-01-21T11:12:29.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: index_shortening: 1 2024-01-21T11:12:29.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: hash_index_allow_collision: 1 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: checksum: 1 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: no_block_cache: 0 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_cache: 0x56029017ef10 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_cache_name: BinnedLRUCache 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_cache_options: 2024-01-21T11:12:29.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: capacity : 536870912 2024-01-21T11:12:29.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: num_shard_bits : 4 2024-01-21T11:12:29.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: strict_capacity_limit : 0 2024-01-21T11:12:29.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: high_pri_pool_ratio: 0.000 2024-01-21T11:12:29.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_cache_compressed: (nil) 2024-01-21T11:12:29.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: persistent_cache: (nil) 2024-01-21T11:12:29.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_size: 4096 2024-01-21T11:12:29.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_size_deviation: 10 2024-01-21T11:12:29.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_restart_interval: 16 2024-01-21T11:12:29.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: index_block_restart_interval: 1 2024-01-21T11:12:29.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: metadata_block_size: 4096 2024-01-21T11:12:29.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: partition_filters: 0 2024-01-21T11:12:29.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: use_delta_encoding: 1 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: whole_key_filtering: 1 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: verify_compression: 0 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: read_amp_bytes_per_bit: 0 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: format_version: 2 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: enable_index_compression: 1 2024-01-21T11:12:29.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: block_align: 0 2024-01-21T11:12:29.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T11:12:29.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T11:12:29.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression: NoCompression 2024-01-21T11:12:29.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T11:12:29.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T11:12:29.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T11:12:29.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.num_levels: 7 2024-01-21T11:12:29.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T11:12:29.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T11:12:29.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T11:12:29.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T11:12:29.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T11:12:29.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T11:12:29.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T11:12:29.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:12:29.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T11:12:29.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T11:12:29.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T11:12:29.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T11:12:29.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T11:12:29.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T11:12:29.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T11:12:29.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T11:12:29.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T11:12:29.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T11:12:29.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T11:12:29.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T11:12:29.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T11:12:29.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T11:12:29.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T11:12:29.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T11:12:29.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T11:12:29.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T11:12:29.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T11:12:29.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T11:12:29.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T11:12:29.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T11:12:29.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T11:12:29.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T11:12:29.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T11:12:29.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T11:12:29.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T11:12:29.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T11:12:29.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T11:12:29.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T11:12:29.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T11:12:29.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T11:12:29.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T11:12:29.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.ttl: 2592000 2024-01-21T11:12:29.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.628+0000 7faa87977700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T11:12:29.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.629+0000 7faa87977700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi089/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T11:12:29.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.629+0000 7faa87977700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T11:12:29.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.629+0000 7faa87977700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705835549629671, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T11:12:29.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.629+0000 7faa87977700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T11:12:29.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.629+0000 7faa87977700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T11:12:29.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.630+0000 7faa87977700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705835549630494, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705835549, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T11:12:29.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.630+0000 7faa87977700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T11:12:29.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.631+0000 7faa87977700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705835549631491, "job": 1, "event": "recovery_finished"} 2024-01-21T11:12:29.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.631+0000 7faa87977700 4 rocksdb: DB pointer 0x560290243800 2024-01-21T11:12:29.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.631+0000 7faa6e78f700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T11:12:29.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.631+0000 7faa6e78f700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T11:12:29.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** DB Stats ** 2024-01-21T11:12:29.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:12:29.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T11:12:29.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T11:12:29.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:12:29.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T11:12:29.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T11:12:29.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:12:29.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** Compaction Stats [default] ** 2024-01-21T11:12:29.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:29.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:29.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** Compaction Stats [default] ** 2024-01-21T11:12:29.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:29.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:29.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:12:29.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:29.980 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:29.980 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:12:29.980 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:12:29.980 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:12:29.980 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Cumulative compaction: 0.00 GB write, 26.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:29.980 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Interval compaction: 0.00 GB write, 26.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:29.981 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:12:29.981 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.981 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:12:29.981 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.981 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** Compaction Stats [default] ** 2024-01-21T11:12:29.981 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** Compaction Stats [default] ** 2024-01-21T11:12:29.982 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:12:29.983 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:12:29.983 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.8 0.00 0.00 1 0.001 0 0 2024-01-21T11:12:29.983 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:12:29.983 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:29.983 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:12:29.983 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:12:29.984 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:12:29.984 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:12:29.984 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Cumulative compaction: 0.00 GB write, 26.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:29.984 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:12:29.984 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:12:29.984 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.985 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:12:29.985 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.985 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.632+0000 7faa87977700 0 starting mon.smithi089 rank 0 at public addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] at bind addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi089 fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:29.985 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 systemd[1]: Started Ceph mon.smithi089 for e44d1d4c-b84d-11ee-95b0-87774f69a715. 2024-01-21T11:12:29.985 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 1 mon.smithi089@-1(???) e1 preinit fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:29.985 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 0 mon.smithi089@-1(???).mds e1 new map 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 0 mon.smithi089@-1(???).mds e1 print_map 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: e1 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: legacy client fscid: -1 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.986 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: No filesystems configured 2024-01-21T11:12:29.987 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: 2024-01-21T11:12:29.987 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 0 mon.smithi089@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T11:12:29.987 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 0 mon.smithi089@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T11:12:29.987 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 0 mon.smithi089@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T11:12:29.987 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 0 mon.smithi089@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T11:12:29.987 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:27.648916+0000 mon.smithi089 (mon.0) 8 : audit [INF] from='client.? 172.21.15.89:0/3246313944' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T11:12:29.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:27.651910+0000 mon.smithi089 (mon.0) 9 : audit [INF] from='client.? 172.21.15.89:0/3246313944' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T11:12:29.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:12:29.633+0000 7faa87977700 1 mon.smithi089@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T11:12:29.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:29.649117+0000 mon.smithi089 (mon.0) 1 : cluster [INF] mon.smithi089 is new leader, mons smithi089 in quorum (ranks 0) 2024-01-21T11:12:29.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:29.649204+0000 mon.smithi089 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} 2024-01-21T11:12:29.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:29.649250+0000 mon.smithi089 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T11:12:29.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:29.649280+0000 mon.smithi089 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T11:12:29.989 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:29.649696+0000 mon.smithi089 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T11:12:29.996 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.954+0000 7f10ca921700 1 Processor -- start 2024-01-21T11:12:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.954+0000 7f10ca921700 1 -- start start 2024-01-21T11:12:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.954+0000 7f10ca921700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c4070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.954+0000 7f10ca921700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40728f0 con 0x7f10c40724d0 2024-01-21T11:12:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c4070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c4070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58752/0 (socket says 172.21.15.89:58752) 2024-01-21T11:12:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c3fff700 1 -- 172.21.15.89:0/4283601782 learned_addr learned my addr 172.21.15.89:0/4283601782 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c3fff700 1 -- 172.21.15.89:0/4283601782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c4070f60 con 0x7f10c40724d0 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c3fff700 1 --2- 172.21.15.89:0/4283601782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c4070a70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f10b800b5f0 tx=0x7f10b8006ee0).ready entity=mon.0 client_cookie=bd725920c6ac760e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c2ffd700 1 -- 172.21.15.89:0/4283601782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10b800d600 con 0x7f10c40724d0 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c2ffd700 1 -- 172.21.15.89:0/4283601782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f10b800dbd0 con 0x7f10c40724d0 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.955+0000 7f10c2ffd700 1 -- 172.21.15.89:0/4283601782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10b8011c00 con 0x7f10c40724d0 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 -- 172.21.15.89:0/4283601782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 msgr2=0x7f10c4070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 --2- 172.21.15.89:0/4283601782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c4070a70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f10b800b5f0 tx=0x7f10b8006ee0).stop 2024-01-21T11:12:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 -- 172.21.15.89:0/4283601782 shutdown_connections 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 --2- 172.21.15.89:0/4283601782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c4070a70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 -- 172.21.15.89:0/4283601782 >> 172.21.15.89:0/4283601782 conn(0x7f10c40f74c0 msgr2=0x7f10c40f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 -- 172.21.15.89:0/4283601782 shutdown_connections 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.956+0000 7f10ca921700 1 -- 172.21.15.89:0/4283601782 wait complete. 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10ca921700 1 Processor -- start 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10ca921700 1 -- start start 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10ca921700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c410cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10ca921700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c410d210 con 0x7f10c40724d0 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c410cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c410cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58762/0 (socket says 172.21.15.89:58762) 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.957+0000 7f10c3fff700 1 -- 172.21.15.89:0/942565350 learned_addr learned my addr 172.21.15.89:0/942565350 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10c3fff700 1 -- 172.21.15.89:0/942565350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b800b200 con 0x7f10c40724d0 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10c3fff700 1 --2- 172.21.15.89:0/942565350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c410cd20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f10b8006c70 tx=0x7f10b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10b800da30 con 0x7f10c40724d0 2024-01-21T11:12:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f10b801a9f0 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c410d410 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10b801b890 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.958+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c410d820 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.959+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f10b800f070 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.959+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f10b80257d0 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.959+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c4045ad0 con 0x7f10c40724d0 2024-01-21T11:12:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:29.961+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f10b8016070 con 0x7f10c40724d0 2024-01-21T11:12:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.024+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f10c410ed50 con 0x7f10c40724d0 2024-01-21T11:12:30.026 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.027+0000 7f10c0ff9700 1 -- 172.21.15.89:0/942565350 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f10b801b9f0 con 0x7f10c40724d0 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.027+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 msgr2=0x7f10c410cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.027+0000 7f10ca921700 1 --2- 172.21.15.89:0/942565350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c410cd20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f10b8006c70 tx=0x7f10b8006ee0).stop 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.027+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 shutdown_connections 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.027+0000 7f10ca921700 1 --2- 172.21.15.89:0/942565350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c40724d0 0x7f10c410cd20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.027+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 >> 172.21.15.89:0/942565350 conn(0x7f10c40f74c0 msgr2=0x7f10c40f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.028+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 shutdown_connections 2024-01-21T11:12:30.028 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:30.028+0000 7f10ca921700 1 -- 172.21.15.89:0/942565350 wait complete. 2024-01-21T11:12:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T11:12:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T11:12:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:Creating mgr... 2024-01-21T11:12:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:Verifying port 9283 ... 2024-01-21T11:12:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl daemon-reload 2024-01-21T11:12:30.454 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw 2024-01-21T11:12:30.476 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl reset-failed ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw 2024-01-21T11:12:30.481 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Failed to reset failed state of unit ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw.service: Unit ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw.service not loaded. 2024-01-21T11:12:30.482 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl enable ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw 2024-01-21T11:12:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715.target.wants/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw.service → /etc/systemd/system/ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@.service. 2024-01-21T11:12:30.572 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mgr.smithi089.ufybqw 2024-01-21T11:12:30.892 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:12:30.900 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: disabled 2024-01-21T11:12:30.901 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:12:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: inactive 2024-01-21T11:12:30.909 INFO:teuthology.orchestra.run.smithi089.stderr:firewalld.service is not enabled 2024-01-21T11:12:30.909 INFO:teuthology.orchestra.run.smithi089.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T11:12:30.909 INFO:teuthology.orchestra.run.smithi089.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T11:12:30.909 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:12:30.916 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: disabled 2024-01-21T11:12:30.917 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:12:30.924 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: inactive 2024-01-21T11:12:30.925 INFO:teuthology.orchestra.run.smithi089.stderr:firewalld.service is not enabled 2024-01-21T11:12:30.925 INFO:teuthology.orchestra.run.smithi089.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T11:12:30.925 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for mgr to start... 2024-01-21T11:12:30.925 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for mgr... 2024-01-21T11:12:30.926 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:12:31.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:31 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:30.027532+0000 mon.smithi089 (mon.0) 6 : audit [INF] from='client.? 172.21.15.89:0/942565350' entity='client.admin' 2024-01-21T11:12:31.291 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.290+0000 7f8fcf408700 1 Processor -- start 2024-01-21T11:12:31.291 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.290+0000 7f8fcf408700 1 -- start start 2024-01-21T11:12:31.291 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.290+0000 7f8fcf408700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc80fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:31.291 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.290+0000 7f8fcf408700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc80fede0 con 0x7f8fc80704a0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fcd1a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc80fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fcd1a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc80fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58778/0 (socket says 172.21.15.89:58778) 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fcd1a4700 1 -- 172.21.15.89:0/4156352598 learned_addr learned my addr 172.21.15.89:0/4156352598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fcd1a4700 1 -- 172.21.15.89:0/4156352598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc80fef20 con 0x7f8fc80704a0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fcd1a4700 1 --2- 172.21.15.89:0/4156352598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc80fe8f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8fb800b5f0 tx=0x7f8fb8006ee0).ready entity=mon.0 client_cookie=9ea9d989c76cee2f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fb7fff700 1 -- 172.21.15.89:0/4156352598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb800d600 con 0x7f8fc80704a0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fb7fff700 1 -- 172.21.15.89:0/4156352598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8fb800dbd0 con 0x7f8fc80704a0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.291+0000 7f8fb7fff700 1 -- 172.21.15.89:0/4156352598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb8011c00 con 0x7f8fc80704a0 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 -- 172.21.15.89:0/4156352598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 msgr2=0x7f8fc80fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 --2- 172.21.15.89:0/4156352598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc80fe8f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8fb800b5f0 tx=0x7f8fb8006ee0).stop 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 -- 172.21.15.89:0/4156352598 shutdown_connections 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 --2- 172.21.15.89:0/4156352598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc80fe8f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:31.295 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 -- 172.21.15.89:0/4156352598 >> 172.21.15.89:0/4156352598 conn(0x7f8fc80f63b0 msgr2=0x7f8fc80f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 -- 172.21.15.89:0/4156352598 shutdown_connections 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.292+0000 7f8fcf408700 1 -- 172.21.15.89:0/4156352598 wait complete. 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcf408700 1 Processor -- start 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcf408700 1 -- start start 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcf408700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc810a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcf408700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc810aed0 con 0x7f8fc80704a0 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcd1a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc810a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcd1a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc810a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58790/0 (socket says 172.21.15.89:58790) 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcd1a4700 1 -- 172.21.15.89:0/1361233170 learned_addr learned my addr 172.21.15.89:0/1361233170 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcd1a4700 1 -- 172.21.15.89:0/1361233170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb800b200 con 0x7f8fc80704a0 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fcd1a4700 1 --2- 172.21.15.89:0/1361233170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc810a9e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8fb800b190 tx=0x7f8fb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:31.296 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.293+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb801b410 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.294+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc810b0d0 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.294+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8fb801b9e0 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.294+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb801a880 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.294+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc810b4e0 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.294+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8fb800f070 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.294+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8fb8025c90 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.295+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc8045ad0 con 0x7f8fc80704a0 2024-01-21T11:12:31.297 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.297+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8fb8016070 con 0x7f8fc80704a0 2024-01-21T11:12:31.367 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.366+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8fc8104800 con 0x7f8fc80704a0 2024-01-21T11:12:31.367 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.366+0000 7f8fb5ffb700 1 -- 172.21.15.89:0/1361233170 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f8fb801bb50 con 0x7f8fc80704a0 2024-01-21T11:12:31.367 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:12:31.367 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "fsid": "e44d1d4c-b84d-11ee-95b0-87774f69a715", 2024-01-21T11:12:31.367 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:12:31.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 0 2024-01-21T11:12:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:12:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "smithi089" 2024-01-21T11:12:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T11:12:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:12:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "available": false, 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:12:26.348365+0000", 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:12:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 msgr2=0x7f8fc810a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 --2- 172.21.15.89:0/1361233170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc810a9e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8fb800b190 tx=0x7f8fb8006ee0).stop 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 shutdown_connections 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 --2- 172.21.15.89:0/1361233170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fc80704a0 0x7f8fc810a9e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 >> 172.21.15.89:0/1361233170 conn(0x7f8fc80f63b0 msgr2=0x7f8fc80f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 shutdown_connections 2024-01-21T11:12:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:31.367+0000 7f8fcf408700 1 -- 172.21.15.89:0/1361233170 wait complete. 2024-01-21T11:12:31.783 INFO:teuthology.orchestra.run.smithi089.stderr:mgr not available, waiting (1/15)... 2024-01-21T11:12:32.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:31.366744+0000 mon.smithi089 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.89:0/1361233170' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:12:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:12:34.142 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.141+0000 7fb5182b9700 1 Processor -- start 2024-01-21T11:12:34.142 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.141+0000 7fb5182b9700 1 -- start start 2024-01-21T11:12:34.143 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb5182b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:34.143 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb5182b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5100fee30 con 0x7fb5100703d0 2024-01-21T11:12:34.143 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb516055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:34.143 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb516055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45462/0 (socket says 172.21.15.89:45462) 2024-01-21T11:12:34.144 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb516055700 1 -- 172.21.15.89:0/1730477934 learned_addr learned my addr 172.21.15.89:0/1730477934 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:34.144 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb516055700 1 -- 172.21.15.89:0/1730477934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5100fef70 con 0x7fb5100703d0 2024-01-21T11:12:34.144 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb516055700 1 --2- 172.21.15.89:0/1730477934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fe940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb50c00b5f0 tx=0x7fb50c006ee0).ready entity=mon.0 client_cookie=2e8f6acb591bb462 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:34.144 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.142+0000 7fb515053700 1 -- 172.21.15.89:0/1730477934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb50c00d600 con 0x7fb5100703d0 2024-01-21T11:12:34.144 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb515053700 1 -- 172.21.15.89:0/1730477934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb50c00dbd0 con 0x7fb5100703d0 2024-01-21T11:12:34.144 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb515053700 1 -- 172.21.15.89:0/1730477934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb50c011c00 con 0x7fb5100703d0 2024-01-21T11:12:34.146 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 -- 172.21.15.89:0/1730477934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 msgr2=0x7fb5100fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 --2- 172.21.15.89:0/1730477934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fe940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb50c00b5f0 tx=0x7fb50c006ee0).stop 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 -- 172.21.15.89:0/1730477934 shutdown_connections 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 --2- 172.21.15.89:0/1730477934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fe940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 -- 172.21.15.89:0/1730477934 >> 172.21.15.89:0/1730477934 conn(0x7fb5100f63c0 msgr2=0x7fb5100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 -- 172.21.15.89:0/1730477934 shutdown_connections 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.143+0000 7fb5182b9700 1 -- 172.21.15.89:0/1730477934 wait complete. 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb5182b9700 1 Processor -- start 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb5182b9700 1 -- start start 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb5182b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb5182b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5100fa840 con 0x7fb5100703d0 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb516055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb516055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45464/0 (socket says 172.21.15.89:45464) 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb516055700 1 -- 172.21.15.89:0/3397809163 learned_addr learned my addr 172.21.15.89:0/3397809163 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:34.147 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb516055700 1 -- 172.21.15.89:0/3397809163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c00b200 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.144+0000 7fb516055700 1 --2- 172.21.15.89:0/3397809163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fc3c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb50c00e040 tx=0x7fb50c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb50c00d9f0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb50c01a9f0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb50c01b8c0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5100fa9e0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5100fadf0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb50c00f070 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.145+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb50c0257d0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.146+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb510045ad0 con 0x7fb5100703d0 2024-01-21T11:12:34.148 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.148+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb50c016070 con 0x7fb5100703d0 2024-01-21T11:12:34.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.216+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb510104820 con 0x7fb5100703d0 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.216+0000 7fb4feffd700 1 -- 172.21.15.89:0/3397809163 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fb50c01ba20 con 0x7fb5100703d0 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "fsid": "e44d1d4c-b84d-11ee-95b0-87774f69a715", 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 0 2024-01-21T11:12:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "smithi089" 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:12:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:12:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "available": false, 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:12:26.348365+0000", 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 msgr2=0x7fb5100fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 --2- 172.21.15.89:0/3397809163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fc3c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb50c00e040 tx=0x7fb50c006ee0).stop 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 shutdown_connections 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 --2- 172.21.15.89:0/3397809163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5100703d0 0x7fb5100fc3c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 >> 172.21.15.89:0/3397809163 conn(0x7fb5100f63c0 msgr2=0x7fb5100f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 shutdown_connections 2024-01-21T11:12:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:34.217+0000 7fb5182b9700 1 -- 172.21.15.89:0/3397809163 wait complete. 2024-01-21T11:12:34.517 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:34.216750+0000 mon.smithi089 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.89:0/3397809163' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:12:34.589 INFO:teuthology.orchestra.run.smithi089.stderr:mgr not available, waiting (2/15)... 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:35.809614+0000 mon.smithi089 (mon.0) 9 : cluster [INF] Activating manager daemon smithi089.ufybqw 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:35.811921+0000 mon.smithi089 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi089.ufybqw(active, starting, since 0.00238531s) 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.812870+0000 mon.smithi089 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.813025+0000 mon.smithi089 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.813129+0000 mon.smithi089 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.813269+0000 mon.smithi089 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.813450+0000 mon.smithi089 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mgr metadata", "who": "smithi089.ufybqw", "id": "smithi089.ufybqw"}]: dispatch 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:35.826939+0000 mon.smithi089 (mon.0) 16 : cluster [INF] Manager daemon smithi089.ufybqw is now available 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.833953+0000 mon.smithi089 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:36.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.837893+0000 mon.smithi089 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:12:36.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.838973+0000 mon.smithi089 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:12:36.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.843567+0000 mon.smithi089 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:36.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:35.845617+0000 mon.smithi089 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.89:0/2056129799' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.933+0000 7f1cd4ec1700 1 Processor -- start 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.933+0000 7f1cd4ec1700 1 -- start start 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.933+0000 7f1cd4ec1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.933+0000 7f1cd4ec1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd00fb490 con 0x7f1cd00fab80 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1cce59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1cce59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45520/0 (socket says 172.21.15.89:45520) 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1cce59c700 1 -- 172.21.15.89:0/462629588 learned_addr learned my addr 172.21.15.89:0/462629588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1cce59c700 1 -- 172.21.15.89:0/462629588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd00fb5d0 con 0x7f1cd00fab80 2024-01-21T11:12:36.938 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1cce59c700 1 --2- 172.21.15.89:0/462629588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd00fafa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1cc400b5f0 tx=0x7f1cc4006ee0).ready entity=mon.0 client_cookie=4624c859192b53d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1ccd59a700 1 -- 172.21.15.89:0/462629588 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cc400d600 con 0x7f1cd00fab80 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1ccd59a700 1 -- 172.21.15.89:0/462629588 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1cc400dbd0 con 0x7f1cd00fab80 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.934+0000 7f1ccd59a700 1 -- 172.21.15.89:0/462629588 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cc4011c00 con 0x7f1cd00fab80 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/462629588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 msgr2=0x7f1cd00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 --2- 172.21.15.89:0/462629588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd00fafa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1cc400b5f0 tx=0x7f1cc4006ee0).stop 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/462629588 shutdown_connections 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 --2- 172.21.15.89:0/462629588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd00fafa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/462629588 >> 172.21.15.89:0/462629588 conn(0x7f1cd00f63c0 msgr2=0x7f1cd00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/462629588 shutdown_connections 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.935+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/462629588 wait complete. 2024-01-21T11:12:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cd4ec1700 1 Processor -- start 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cd4ec1700 1 -- start start 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cd4ec1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd0109920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cd4ec1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd0109e10 con 0x7f1cd00fab80 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cce59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd0109920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cce59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd0109920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45524/0 (socket says 172.21.15.89:45524) 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cce59c700 1 -- 172.21.15.89:0/305136295 learned_addr learned my addr 172.21.15.89:0/305136295 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cce59c700 1 -- 172.21.15.89:0/305136295 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc400b200 con 0x7f1cd00fab80 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cce59c700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd0109920 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1cc400e040 tx=0x7f1cc4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.936+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cc4011a20 con 0x7f1cd00fab80 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1cc400d9f0 con 0x7f1cd00fab80 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cc4022790 con 0x7f1cd00fab80 2024-01-21T11:12:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd010a010 con 0x7f1cd00fab80 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd010a420 con 0x7f1cd00fab80 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f1cc4029060 con 0x7f1cd00fab80 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cbeffd700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f1cb402fc70 0x7f1cb4032130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.937+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1cc4042790 con 0x7f1cd00fab80 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.938+0000 7f1ccdd9b700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f1cb402fc70 0x7f1cb4032130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.938+0000 7f1ccdd9b700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f1cb402fc70 0x7f1cb4032130 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1cc00096a0 tx=0x7f1cc0006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.938+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd0059100 con 0x7f1cd00fab80 2024-01-21T11:12:36.942 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:36.943+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cc4016070 con 0x7f1cd00fab80 2024-01-21T11:12:37.151 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.144+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1cd0045ad0 con 0x7f1cd00fab80 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.145+0000 7f1cbeffd700 1 -- 172.21.15.89:0/305136295 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f1cc402c440 con 0x7f1cd00fab80 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "fsid": "e44d1d4c-b84d-11ee-95b0-87774f69a715", 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 0 2024-01-21T11:12:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "smithi089" 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:12:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:12:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "available": true, 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ], 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:12:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:12:26.348365+0000", 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: }, 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f1cb402fc70 msgr2=0x7f1cb4032130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f1cb402fc70 0x7f1cb4032130 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1cc00096a0 tx=0x7f1cc0006b40).stop 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 msgr2=0x7f1cd0109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd0109920 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1cc400e040 tx=0x7f1cc4006ee0).stop 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 shutdown_connections 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f1cb402fc70 0x7f1cb4032130 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 --2- 172.21.15.89:0/305136295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd00fab80 0x7f1cd0109920 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 >> 172.21.15.89:0/305136295 conn(0x7f1cd00f63c0 msgr2=0x7f1cd00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 shutdown_connections 2024-01-21T11:12:37.156 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.148+0000 7f1cd4ec1700 1 -- 172.21.15.89:0/305136295 wait complete. 2024-01-21T11:12:37.505 INFO:teuthology.orchestra.run.smithi089.stderr:mgr is available 2024-01-21T11:12:37.505 INFO:teuthology.orchestra.run.smithi089.stderr:Enabling cephadm module... 2024-01-21T11:12:37.506 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T11:12:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.832+0000 7f6bf9f9b700 1 Processor -- start 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.832+0000 7f6bf9f9b700 1 -- start start 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf9f9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf9f9b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40fc5c0 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45538/0 (socket says 172.21.15.89:45538) 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf37fe700 1 -- 172.21.15.89:0/1372437792 learned_addr learned my addr 172.21.15.89:0/1372437792 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf37fe700 1 -- 172.21.15.89:0/1372437792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf40fc700 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf37fe700 1 --2- 172.21.15.89:0/1372437792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40fc0d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6bec00b5f0 tx=0x7f6bec006ee0).ready entity=mon.0 client_cookie=56ddaa050a151425 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf27fc700 1 -- 172.21.15.89:0/1372437792 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bec00d600 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf27fc700 1 -- 172.21.15.89:0/1372437792 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6bec00dbd0 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.833+0000 7f6bf27fc700 1 -- 172.21.15.89:0/1372437792 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bec011c00 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/1372437792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 msgr2=0x7f6bf40fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 --2- 172.21.15.89:0/1372437792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40fc0d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6bec00b5f0 tx=0x7f6bec006ee0).stop 2024-01-21T11:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/1372437792 shutdown_connections 2024-01-21T11:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 --2- 172.21.15.89:0/1372437792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40fc0d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/1372437792 >> 172.21.15.89:0/1372437792 conn(0x7f6bf40f74b0 msgr2=0x7f6bf40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/1372437792 shutdown_connections 2024-01-21T11:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.834+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/1372437792 wait complete. 2024-01-21T11:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf9f9b700 1 Processor -- start 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf9f9b700 1 -- start start 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf9f9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf9f9b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4070a60 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45542/0 (socket says 172.21.15.89:45542) 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf37fe700 1 -- 172.21.15.89:0/3136929824 learned_addr learned my addr 172.21.15.89:0/3136929824 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.835+0000 7f6bf37fe700 1 -- 172.21.15.89:0/3136929824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec00b200 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.836+0000 7f6bf37fe700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40725e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6bec00b1d0 tx=0x7f6bec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.836+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bec01b410 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.836+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6bec01b9e0 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.836+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/3136929824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf4070c00 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.836+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bec01a840 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.836+0000 7f6bf9f9b700 1 -- 172.21.15.89:0/3136929824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf4071010 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.837+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f6bec00f070 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.837+0000 7f6bdbfff700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f6be002fc50 0x7f6be0032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.837+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6bec041260 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.837+0000 7f6bf2ffd700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f6be002fc50 0x7f6be0032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.837+0000 7f6bf2ffd700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f6be002fc50 0x7f6be0032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6be80096a0 tx=0x7f6be8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.837+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc004fa0 con 0x7f6bf40fbcb0 2024-01-21T11:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:37.841+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bec016070 con 0x7f6bf40fbcb0 2024-01-21T11:12:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.026+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f6bcc005670 con 0x7f6bf40fbcb0 2024-01-21T11:12:38.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:37 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:36.814546+0000 mon.smithi089 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi089.ufybqw(active, since 1.00501s) 2024-01-21T11:12:38.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:37 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:37.145703+0000 mon.smithi089 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.89:0/305136295' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:12:38.813 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.812+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6bec01bb50 con 0x7f6bf40fbcb0 2024-01-21T11:12:38.815 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.814+0000 7f6bdbfff700 1 -- 172.21.15.89:0/3136929824 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f6bec0237d0 con 0x7f6bf40fbcb0 2024-01-21T11:12:38.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.816+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f6be002fc50 msgr2=0x7f6be0032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.816+0000 7f6bd9ffb700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f6be002fc50 0x7f6be0032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6be80096a0 tx=0x7f6be8006b40).stop 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.816+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 msgr2=0x7f6bf40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.816+0000 7f6bd9ffb700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40725e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6bec00b1d0 tx=0x7f6bec006ee0).stop 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.817+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 shutdown_connections 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.817+0000 7f6bd9ffb700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f6be002fc50 0x7f6be0032110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.817+0000 7f6bd9ffb700 1 --2- 172.21.15.89:0/3136929824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf40fbcb0 0x7f6bf40725e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.817+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 >> 172.21.15.89:0/3136929824 conn(0x7f6bf40f74b0 msgr2=0x7f6bf40f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.817+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 shutdown_connections 2024-01-21T11:12:38.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:38.817+0000 7f6bd9ffb700 1 -- 172.21.15.89:0/3136929824 wait complete. 2024-01-21T11:12:39.116 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:38.027084+0000 mon.smithi089 (mon.0) 24 : audit [INF] from='client.? 172.21.15.89:0/3136929824' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T11:12:39.181 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.527+0000 7f255d4f7700 1 Processor -- start 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.527+0000 7f255d4f7700 1 -- start start 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.527+0000 7f255d4f7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.527+0000 7f255d4f7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25580fedd0 con 0x7f2558070430 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.527+0000 7f2556ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2556ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45576/0 (socket says 172.21.15.89:45576) 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2556ffd700 1 -- 172.21.15.89:0/1027407144 learned_addr learned my addr 172.21.15.89:0/1027407144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2556ffd700 1 -- 172.21.15.89:0/1027407144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580fef10 con 0x7f2558070430 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2556ffd700 1 --2- 172.21.15.89:0/1027407144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fe8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f254800b5f0 tx=0x7f2548006ee0).ready entity=mon.0 client_cookie=76bf8ba7d7d7189d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:39.530 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2555ffb700 1 -- 172.21.15.89:0/1027407144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254800d600 con 0x7f2558070430 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2555ffb700 1 -- 172.21.15.89:0/1027407144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f254800dbd0 con 0x7f2558070430 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.528+0000 7f2555ffb700 1 -- 172.21.15.89:0/1027407144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2548011c00 con 0x7f2558070430 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 -- 172.21.15.89:0/1027407144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 msgr2=0x7f25580fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 --2- 172.21.15.89:0/1027407144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fe8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f254800b5f0 tx=0x7f2548006ee0).stop 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 -- 172.21.15.89:0/1027407144 shutdown_connections 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 --2- 172.21.15.89:0/1027407144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fe8e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:39.531 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 -- 172.21.15.89:0/1027407144 >> 172.21.15.89:0/1027407144 conn(0x7f25580f63c0 msgr2=0x7f25580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:39.533 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 -- 172.21.15.89:0/1027407144 shutdown_connections 2024-01-21T11:12:39.533 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.529+0000 7f255d4f7700 1 -- 172.21.15.89:0/1027407144 wait complete. 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f255d4f7700 1 Processor -- start 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f255d4f7700 1 -- start start 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f255d4f7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f255d4f7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25580fa7e0 con 0x7f2558070430 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f2556ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f2556ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45588/0 (socket says 172.21.15.89:45588) 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.530+0000 7f2556ffd700 1 -- 172.21.15.89:0/3457026279 learned_addr learned my addr 172.21.15.89:0/3457026279 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f2556ffd700 1 -- 172.21.15.89:0/3457026279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254800b200 con 0x7f2558070430 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f2556ffd700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fc360 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f254800e040 tx=0x7f2548006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254800d9f0 con 0x7f2558070430 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25580fa980 con 0x7f2558070430 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f254801a9f0 con 0x7f2558070430 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254801b8c0 con 0x7f2558070430 2024-01-21T11:12:39.534 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.531+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25580fad90 con 0x7f2558070430 2024-01-21T11:12:39.535 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.532+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f254800f070 con 0x7f2558070430 2024-01-21T11:12:39.535 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.532+0000 7f253f7fe700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 0x7f2544032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:39.535 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.532+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f25480412e0 con 0x7f2558070430 2024-01-21T11:12:39.535 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.532+0000 7f25567fc700 1 -- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 msgr2=0x7f2544032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:39.535 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.532+0000 7f25567fc700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 0x7f2544032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:12:39.535 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.532+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2558059100 con 0x7f2558070430 2024-01-21T11:12:39.537 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.537+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2548016070 con 0x7f2558070430 2024-01-21T11:12:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.733+0000 7f25567fc700 1 -- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 msgr2=0x7f2544032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.733+0000 7f25567fc700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 0x7f2544032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.751+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f25580ffb90 con 0x7f2558070430 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.751+0000 7f253f7fe700 1 -- 172.21.15.89:0/3457026279 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f2548025460 con 0x7f2558070430 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 4, 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "available": true, 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "active_name": "smithi089.ufybqw", 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T11:12:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 msgr2=0x7f2544032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 0x7f2544032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 msgr2=0x7f25580fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fc360 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f254800e040 tx=0x7f2548006ee0).stop 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 shutdown_connections 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f254402fd50 0x7f2544032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:12:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 --2- 172.21.15.89:0/3457026279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558070430 0x7f25580fc360 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:39.756 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.753+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 >> 172.21.15.89:0/3457026279 conn(0x7f25580f63c0 msgr2=0x7f25580f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:39.756 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.754+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 shutdown_connections 2024-01-21T11:12:39.756 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:39.754+0000 7f255d4f7700 1 -- 172.21.15.89:0/3457026279 wait complete. 2024-01-21T11:12:40.032 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:38.815743+0000 mon.smithi089 (mon.0) 25 : audit [INF] from='client.? 172.21.15.89:0/3136929824' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T11:12:40.033 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:38.815803+0000 mon.smithi089 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi089.ufybqw(active, since 3s) 2024-01-21T11:12:40.033 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:39.753198+0000 mon.smithi089 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.89:0/3457026279' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T11:12:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for the mgr to restart... 2024-01-21T11:12:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for mgr epoch 4... 2024-01-21T11:12:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T11:12:40.478 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.474+0000 7f70c2029700 1 Processor -- start 2024-01-21T11:12:40.478 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.474+0000 7f70c2029700 1 -- start start 2024-01-21T11:12:40.478 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.474+0000 7f70c2029700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.474+0000 7f70c2029700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0fedd0 con 0x7f70bc070430 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70bb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70bb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45592/0 (socket says 172.21.15.89:45592) 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70bb7fe700 1 -- 172.21.15.89:0/2764356788 learned_addr learned my addr 172.21.15.89:0/2764356788 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70bb7fe700 1 -- 172.21.15.89:0/2764356788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc0fef10 con 0x7f70bc070430 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70bb7fe700 1 --2- 172.21.15.89:0/2764356788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc0fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f70b000b5f0 tx=0x7f70b0006ee0).ready entity=mon.0 client_cookie=f6d6e431d0e75bc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70ba7fc700 1 -- 172.21.15.89:0/2764356788 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b000d600 con 0x7f70bc070430 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70ba7fc700 1 -- 172.21.15.89:0/2764356788 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f70b000dbd0 con 0x7f70bc070430 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.475+0000 7f70ba7fc700 1 -- 172.21.15.89:0/2764356788 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b0011c00 con 0x7f70bc070430 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 -- 172.21.15.89:0/2764356788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 msgr2=0x7f70bc0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 --2- 172.21.15.89:0/2764356788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc0fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f70b000b5f0 tx=0x7f70b0006ee0).stop 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 -- 172.21.15.89:0/2764356788 shutdown_connections 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 --2- 172.21.15.89:0/2764356788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc0fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 -- 172.21.15.89:0/2764356788 >> 172.21.15.89:0/2764356788 conn(0x7f70bc0f63c0 msgr2=0x7f70bc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:40.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 -- 172.21.15.89:0/2764356788 shutdown_connections 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 -- 172.21.15.89:0/2764356788 wait complete. 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.476+0000 7f70c2029700 1 Processor -- start 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70c2029700 1 -- start start 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70c2029700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70c2029700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc10aef0 con 0x7f70bc070430 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70bb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70bb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45598/0 (socket says 172.21.15.89:45598) 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70bb7fe700 1 -- 172.21.15.89:0/2463397007 learned_addr learned my addr 172.21.15.89:0/2463397007 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70bb7fe700 1 -- 172.21.15.89:0/2463397007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b000b200 con 0x7f70bc070430 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70bb7fe700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc10aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f70b000b190 tx=0x7f70b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.477+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b001b410 con 0x7f70bc070430 2024-01-21T11:12:40.481 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.478+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f70b001b9e0 con 0x7f70bc070430 2024-01-21T11:12:40.481 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.478+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b001a880 con 0x7f70bc070430 2024-01-21T11:12:40.481 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.478+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc10b0f0 con 0x7f70bc070430 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.478+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc10b500 con 0x7f70bc070430 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.479+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f70b000f070 con 0x7f70bc070430 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.479+0000 7f70a3fff700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.479+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 --> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70bc059100 con 0x7f70a802fd50 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.479+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f70b00411d0 con 0x7f70bc070430 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.479+0000 7f70baffd700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 msgr2=0x7f70a8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.479+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:12:40.681 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.680+0000 7f70baffd700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 msgr2=0x7f70a8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:40.681 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:40.680+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:12:41.082 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:41.081+0000 7f70baffd700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 msgr2=0x7f70a8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:41.082 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:41.081+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T11:12:41.883 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:41.882+0000 7f70baffd700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 msgr2=0x7f70a8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:41.883 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:41.882+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T11:12:43.484 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:43.483+0000 7f70baffd700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 msgr2=0x7f70a8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3313665215 2024-01-21T11:12:43.485 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:43.483+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T11:12:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:43.815+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f70b0025980 con 0x7f70bc070430 2024-01-21T11:12:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:43.815+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 msgr2=0x7f70a8032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:12:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:43.815+0000 7f70a3fff700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:12:44.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:43.764405+0000 mon.smithi089 (mon.0) 28 : cluster [INF] Active manager daemon smithi089.ufybqw restarted 2024-01-21T11:12:44.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:43.764829+0000 mon.smithi089 (mon.0) 29 : cluster [INF] Activating manager daemon smithi089.ufybqw 2024-01-21T11:12:44.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:43.766581+0000 mon.smithi089 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T11:12:44.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.821+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f70b0042910 con 0x7f70bc070430 2024-01-21T11:12:44.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.822+0000 7f70a3fff700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f70a8033670 0x7f70a8035a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:44.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.822+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70bc059100 con 0x7f70a8033670 2024-01-21T11:12:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.822+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f70a8033670 0x7f70a8035a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.823+0000 7f70baffd700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f70a8033670 0x7f70a8035a60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70ac006fd0 tx=0x7f70ac006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.824+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f70bc059100 con 0x7f70a8033670 2024-01-21T11:12:44.831 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f70bc045ad0 con 0x7f70a8033670 2024-01-21T11:12:44.832 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70a3fff700 1 -- 172.21.15.89:0/2463397007 <== mgr.14114 v2:172.21.15.89:6800/4200893972 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f70bc045ad0 con 0x7f70a8033670 2024-01-21T11:12:44.832 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:12:44.832 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2024-01-21T11:12:44.832 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T11:12:44.832 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f70a8033670 msgr2=0x7f70a8035a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f70a8033670 0x7f70a8035a60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70ac006fd0 tx=0x7f70ac006cf0).stop 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 msgr2=0x7f70bc10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc10aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f70b000b190 tx=0x7f70b0006ee0).stop 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 shutdown_connections 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f70a8033670 0x7f70a8035a60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70bc070430 0x7f70bc10aa00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 --2- 172.21.15.89:0/2463397007 >> [v2:172.21.15.89:6800/3313665215,v1:172.21.15.89:6801/3313665215] conn(0x7f70a802fd50 0x7f70a8032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:12:44.834 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.831+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 >> 172.21.15.89:0/2463397007 conn(0x7f70bc0f63c0 msgr2=0x7f70bc0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:44.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.832+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 shutdown_connections 2024-01-21T11:12:44.835 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:44.832+0000 7f70c2029700 1 -- 172.21.15.89:0/2463397007 wait complete. 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:43.817167+0000 mon.smithi089 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi089.ufybqw(active, starting, since 0.0524072s) 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.819058+0000 mon.smithi089 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.819591+0000 mon.smithi089 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mgr metadata", "who": "smithi089.ufybqw", "id": "smithi089.ufybqw"}]: dispatch 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.820232+0000 mon.smithi089 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.820347+0000 mon.smithi089 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.820467+0000 mon.smithi089 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:43.833921+0000 mon.smithi089 (mon.0) 37 : cluster [INF] Manager daemon smithi089.ufybqw is now available 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.849158+0000 mon.smithi089 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.854366+0000 mon.smithi089 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:45.112 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.859211+0000 mon.smithi089 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:45.113 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.949131+0000 mon.smithi089 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:45.113 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.949791+0000 mon.smithi089 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:12:45.113 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.951408+0000 mon.smithi089 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:12:45.113 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.953181+0000 mon.smithi089 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:12:45.113 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.962311+0000 mon.smithi089 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:12:45.113 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:43.963428+0000 mon.smithi089 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:12:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:mgr epoch 4 is available 2024-01-21T11:12:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:Setting orchestrator backend to cephadm... 2024-01-21T11:12:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T11:12:45.562 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e72f4c700 1 Processor -- start 2024-01-21T11:12:45.562 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e72f4c700 1 -- start start 2024-01-21T11:12:45.562 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e72f4c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e72f4c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0fee10 con 0x7f2e6c0703f0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e70ce8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e70ce8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50684/0 (socket says 172.21.15.89:50684) 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.559+0000 7f2e70ce8700 1 -- 172.21.15.89:0/2684453012 learned_addr learned my addr 172.21.15.89:0/2684453012 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e70ce8700 1 -- 172.21.15.89:0/2684453012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c0fef50 con 0x7f2e6c0703f0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e70ce8700 1 --2- 172.21.15.89:0/2684453012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fe920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2e6000b5f0 tx=0x7f2e60006ee0).ready entity=mon.0 client_cookie=dd9d13b4f080c521 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e6b7fe700 1 -- 172.21.15.89:0/2684453012 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6000d600 con 0x7f2e6c0703f0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e6b7fe700 1 -- 172.21.15.89:0/2684453012 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2e6000dbd0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e6b7fe700 1 -- 172.21.15.89:0/2684453012 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e60011c00 con 0x7f2e6c0703f0 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e72f4c700 1 -- 172.21.15.89:0/2684453012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 msgr2=0x7f2e6c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e72f4c700 1 --2- 172.21.15.89:0/2684453012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fe920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2e6000b5f0 tx=0x7f2e60006ee0).stop 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e72f4c700 1 -- 172.21.15.89:0/2684453012 shutdown_connections 2024-01-21T11:12:45.563 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.560+0000 7f2e72f4c700 1 --2- 172.21.15.89:0/2684453012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fe920 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:45.564 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.561+0000 7f2e72f4c700 1 -- 172.21.15.89:0/2684453012 >> 172.21.15.89:0/2684453012 conn(0x7f2e6c0f63c0 msgr2=0x7f2e6c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:45.564 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.561+0000 7f2e72f4c700 1 -- 172.21.15.89:0/2684453012 shutdown_connections 2024-01-21T11:12:45.564 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.561+0000 7f2e72f4c700 1 -- 172.21.15.89:0/2684453012 wait complete. 2024-01-21T11:12:45.564 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.561+0000 7f2e72f4c700 1 Processor -- start 2024-01-21T11:12:45.564 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.561+0000 7f2e72f4c700 1 -- start start 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e72f4c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e72f4c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0fa820 con 0x7f2e6c0703f0 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e70ce8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e70ce8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50698/0 (socket says 172.21.15.89:50698) 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e70ce8700 1 -- 172.21.15.89:0/3684239198 learned_addr learned my addr 172.21.15.89:0/3684239198 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e70ce8700 1 -- 172.21.15.89:0/3684239198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6000b200 con 0x7f2e6c0703f0 2024-01-21T11:12:45.566 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e70ce8700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fc3a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2e6000e040 tx=0x7f2e60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6000d9f0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c0fa9c0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2e6001a9f0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.562+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6001b8c0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.563+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c0fadd0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.563+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c045ad0 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.564+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f2e6000f070 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.564+0000 7f2e697fa700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f2e5002fcc0 0x7f2e50032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.564+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2e60041990 con 0x7f2e6c0703f0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.564+0000 7f2e6bfff700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f2e5002fcc0 0x7f2e50032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.564+0000 7f2e6bfff700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f2e5002fcc0 0x7f2e50032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2e580096a0 tx=0x7f2e58006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.568+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e60019d60 con 0x7f2e6c0703f0 2024-01-21T11:12:45.745 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.744+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e6c0fb010 con 0x7f2e5002fcc0 2024-01-21T11:12:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.749+0000 7f2e697fa700 1 -- 172.21.15.89:0/3684239198 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2e6c0fb010 con 0x7f2e5002fcc0 2024-01-21T11:12:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f2e5002fcc0 msgr2=0x7f2e50032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f2e5002fcc0 0x7f2e50032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2e580096a0 tx=0x7f2e58006b40).stop 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 msgr2=0x7f2e6c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fc3a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2e6000e040 tx=0x7f2e60006ee0).stop 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 shutdown_connections 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f2e5002fcc0 0x7f2e50032180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 --2- 172.21.15.89:0/3684239198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c0703f0 0x7f2e6c0fc3a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 >> 172.21.15.89:0/3684239198 conn(0x7f2e6c0f63c0 msgr2=0x7f2e6c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 shutdown_connections 2024-01-21T11:12:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:45.751+0000 7f2e72f4c700 1 -- 172.21.15.89:0/3684239198 wait complete. 2024-01-21T11:12:46.040 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:44.825240+0000 mgr.smithi089.ufybqw (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T11:12:46.040 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:44.825927+0000 mon.smithi089 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi089.ufybqw(active, since 1.06116s) 2024-01-21T11:12:46.040 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:44.832704+0000 mgr.smithi089.ufybqw (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T11:12:46.040 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:45.750597+0000 mon.smithi089 (mon.0) 48 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:46.040 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:45.754063+0000 mon.smithi089 (mon.0) 49 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:12:46.087 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T11:12:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.412+0000 7f69759b2700 1 Processor -- start 2024-01-21T11:12:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.412+0000 7f69759b2700 1 -- start start 2024-01-21T11:12:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f69759b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f69700fd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f69759b2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700fdc40 con 0x7f69700fd330 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f69700fd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f69700fd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50714/0 (socket says 172.21.15.89:50714) 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696effd700 1 -- 172.21.15.89:0/3502738589 learned_addr learned my addr 172.21.15.89:0/3502738589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696effd700 1 -- 172.21.15.89:0/3502738589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700fdd80 con 0x7f69700fd330 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696effd700 1 --2- 172.21.15.89:0/3502738589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f69700fd750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f696400b5f0 tx=0x7f6964006ee0).ready entity=mon.0 client_cookie=3475cbe9f16668c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696dffb700 1 -- 172.21.15.89:0/3502738589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696400d600 con 0x7f69700fd330 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696dffb700 1 -- 172.21.15.89:0/3502738589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f696400dbd0 con 0x7f69700fd330 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.413+0000 7f696dffb700 1 -- 172.21.15.89:0/3502738589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6964011c00 con 0x7f69700fd330 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 -- 172.21.15.89:0/3502738589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 msgr2=0x7f69700fd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 --2- 172.21.15.89:0/3502738589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f69700fd750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f696400b5f0 tx=0x7f6964006ee0).stop 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 -- 172.21.15.89:0/3502738589 shutdown_connections 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 --2- 172.21.15.89:0/3502738589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f69700fd750 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 -- 172.21.15.89:0/3502738589 >> 172.21.15.89:0/3502738589 conn(0x7f69700f5b10 msgr2=0x7f69700f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 -- 172.21.15.89:0/3502738589 shutdown_connections 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.414+0000 7f69759b2700 1 -- 172.21.15.89:0/3502738589 wait complete. 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69759b2700 1 Processor -- start 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69759b2700 1 -- start start 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69759b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f697010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69759b2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697010af00 con 0x7f69700fd330 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f696effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f697010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f696effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f697010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50720/0 (socket says 172.21.15.89:50720) 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f696effd700 1 -- 172.21.15.89:0/1181691209 learned_addr learned my addr 172.21.15.89:0/1181691209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f696effd700 1 -- 172.21.15.89:0/1181691209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696400b200 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f696effd700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f697010aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6964002f40 tx=0x7f6964006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696401b410 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697010b100 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f696401b9e0 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696401a8b0 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.415+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697010b510 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.416+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f696400f070 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.416+0000 7f69577fe700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f695c02fcc0 0x7f695c032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.416+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6970045ad0 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.416+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f69640410f0 con 0x7f69700fd330 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.416+0000 7f696e7fc700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f695c02fcc0 0x7f695c032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.417+0000 7f696e7fc700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f695c02fcc0 0x7f695c032180 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f69600096a0 tx=0x7f6960006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.421+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6964016070 con 0x7f69700fd330 2024-01-21T11:12:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.571+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f697010bdf0 con 0x7f695c02fcc0 2024-01-21T11:12:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.572+0000 7f69577fe700 1 -- 172.21.15.89:0/1181691209 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f697010bdf0 con 0x7f695c02fcc0 2024-01-21T11:12:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: value unchanged 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f695c02fcc0 msgr2=0x7f695c032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f695c02fcc0 0x7f695c032180 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f69600096a0 tx=0x7f6960006b40).stop 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 msgr2=0x7f697010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f697010aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6964002f40 tx=0x7f6964006ee0).stop 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 shutdown_connections 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f695c02fcc0 0x7f695c032180 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 --2- 172.21.15.89:0/1181691209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69700fd330 0x7f697010aa10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 >> 172.21.15.89:0/1181691209 conn(0x7f69700f5b10 msgr2=0x7f69700ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.573+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 shutdown_connections 2024-01-21T11:12:46.576 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:46.574+0000 7f69759b2700 1 -- 172.21.15.89:0/1181691209 wait complete. 2024-01-21T11:12:46.898 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:45.746699+0000 mgr.smithi089.ufybqw (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:46.898 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:46.751374+0000 mon.smithi089 (mon.0) 50 : cluster [DBG] mgrmap e7: smithi089.ufybqw(active, since 2s) 2024-01-21T11:12:46.963 INFO:teuthology.orchestra.run.smithi089.stderr:Generating ssh key... 2024-01-21T11:12:46.963 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.294+0000 7f5b29380700 1 Processor -- start 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.294+0000 7f5b29380700 1 -- start start 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b29380700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b240fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b29380700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240fee20 con 0x7f5b24070460 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b22ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b240fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b22ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b240fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50736/0 (socket says 172.21.15.89:50736) 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b22ffd700 1 -- 172.21.15.89:0/2270758328 learned_addr learned my addr 172.21.15.89:0/2270758328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:47.300 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b22ffd700 1 -- 172.21.15.89:0/2270758328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240fef60 con 0x7f5b24070460 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b22ffd700 1 --2- 172.21.15.89:0/2270758328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b240fe930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5b1800b5f0 tx=0x7f5b18006ee0).ready entity=mon.0 client_cookie=41a15f900c8d998b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b21ffb700 1 -- 172.21.15.89:0/2270758328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1800d600 con 0x7f5b24070460 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b21ffb700 1 -- 172.21.15.89:0/2270758328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5b1800dbd0 con 0x7f5b24070460 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.295+0000 7f5b21ffb700 1 -- 172.21.15.89:0/2270758328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b18011c00 con 0x7f5b24070460 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 -- 172.21.15.89:0/2270758328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 msgr2=0x7f5b240fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 --2- 172.21.15.89:0/2270758328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b240fe930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5b1800b5f0 tx=0x7f5b18006ee0).stop 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 -- 172.21.15.89:0/2270758328 shutdown_connections 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 --2- 172.21.15.89:0/2270758328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b240fe930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 -- 172.21.15.89:0/2270758328 >> 172.21.15.89:0/2270758328 conn(0x7f5b240f63b0 msgr2=0x7f5b240f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 -- 172.21.15.89:0/2270758328 shutdown_connections 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.296+0000 7f5b29380700 1 -- 172.21.15.89:0/2270758328 wait complete. 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b29380700 1 Processor -- start 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b29380700 1 -- start start 2024-01-21T11:12:47.301 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b29380700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b2410a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b29380700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2410aec0 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b22ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b2410a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b22ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b2410a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50740/0 (socket says 172.21.15.89:50740) 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b22ffd700 1 -- 172.21.15.89:0/1024393582 learned_addr learned my addr 172.21.15.89:0/1024393582 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.297+0000 7f5b22ffd700 1 -- 172.21.15.89:0/1024393582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1800b200 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b22ffd700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b2410a9d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5b1800b190 tx=0x7f5b18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1801b410 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5b1801b9e0 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2410b0c0 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1801a880 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2410b4d0 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.298+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5b1800f070 con 0x7f5b24070460 2024-01-21T11:12:47.302 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.299+0000 7f5b0b7fe700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f5b1002fdb0 0x7f5b10032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:47.303 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.299+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5b18041190 con 0x7f5b24070460 2024-01-21T11:12:47.303 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.299+0000 7f5b227fc700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f5b1002fdb0 0x7f5b10032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:47.303 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.299+0000 7f5b227fc700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f5b1002fdb0 0x7f5b10032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5b140096a0 tx=0x7f5b14006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:47.303 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.299+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24059100 con 0x7f5b24070460 2024-01-21T11:12:47.304 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.303+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b18016070 con 0x7f5b24070460 2024-01-21T11:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.452+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b24104900 con 0x7f5b1002fdb0 2024-01-21T11:12:47.569 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.568+0000 7f5b0b7fe700 1 -- 172.21.15.89:0/1024393582 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b24104900 con 0x7f5b1002fdb0 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f5b1002fdb0 msgr2=0x7f5b10032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f5b1002fdb0 0x7f5b10032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5b140096a0 tx=0x7f5b14006b40).stop 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 msgr2=0x7f5b2410a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b2410a9d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5b1800b190 tx=0x7f5b18006ee0).stop 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 shutdown_connections 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f5b1002fdb0 0x7f5b10032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 --2- 172.21.15.89:0/1024393582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b24070460 0x7f5b2410a9d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 >> 172.21.15.89:0/1024393582 conn(0x7f5b240f63b0 msgr2=0x7f5b240f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 shutdown_connections 2024-01-21T11:12:47.572 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:47.570+0000 7f5b29380700 1 -- 172.21.15.89:0/1024393582 wait complete. 2024-01-21T11:12:47.896 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T11:12:48.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:46.573548+0000 mgr.smithi089.ufybqw (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:48.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:47.567400+0000 mon.smithi089 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:48.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:47.569187+0000 mon.smithi089 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:49.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:12:47.454652+0000 mgr.smithi089.ufybqw (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff416ec2700 1 Processor -- start 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff416ec2700 1 -- start start 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff416ec2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff410100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff416ec2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100fc750 con 0x7ff4100703b0 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff414c5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff410100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff414c5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff410100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50754/0 (socket says 172.21.15.89:50754) 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff414c5e700 1 -- 172.21.15.89:0/71435737 learned_addr learned my addr 172.21.15.89:0/71435737 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.246+0000 7ff414c5e700 1 -- 172.21.15.89:0/71435737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410101250 con 0x7ff4100703b0 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff414c5e700 1 --2- 172.21.15.89:0/71435737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff410100d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ff40400b5f0 tx=0x7ff404006ee0).ready entity=mon.0 client_cookie=56c4b6b6be5dd580 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff40f7fe700 1 -- 172.21.15.89:0/71435737 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40400d600 con 0x7ff4100703b0 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff40f7fe700 1 -- 172.21.15.89:0/71435737 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff40400dbd0 con 0x7ff4100703b0 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff40f7fe700 1 -- 172.21.15.89:0/71435737 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff404011c00 con 0x7ff4100703b0 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 -- 172.21.15.89:0/71435737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 msgr2=0x7ff410100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 --2- 172.21.15.89:0/71435737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff410100d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ff40400b5f0 tx=0x7ff404006ee0).stop 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 -- 172.21.15.89:0/71435737 shutdown_connections 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 --2- 172.21.15.89:0/71435737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff410100d60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 -- 172.21.15.89:0/71435737 >> 172.21.15.89:0/71435737 conn(0x7ff4100f63c0 msgr2=0x7ff4100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 -- 172.21.15.89:0/71435737 shutdown_connections 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.247+0000 7ff416ec2700 1 -- 172.21.15.89:0/71435737 wait complete. 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff416ec2700 1 Processor -- start 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff416ec2700 1 -- start start 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff416ec2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff41010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff416ec2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41010d100 con 0x7ff4100703b0 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff414c5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff41010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:49.282 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff414c5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff41010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50766/0 (socket says 172.21.15.89:50766) 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.248+0000 7ff414c5e700 1 -- 172.21.15.89:0/3207589274 learned_addr learned my addr 172.21.15.89:0/3207589274 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff414c5e700 1 -- 172.21.15.89:0/3207589274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40400b200 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff414c5e700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff41010cc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff40400b190 tx=0x7ff404006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40401b410 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff40401b9e0 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41010d300 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40401a880 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41010d710 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.249+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff40400f070 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.250+0000 7ff40d7fa700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7ff3f402fd50 0x7ff3f4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.250+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff404041270 con 0x7ff4100703b0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.250+0000 7ff40ffff700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7ff3f402fd50 0x7ff3f4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:49.283 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.250+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff410059100 con 0x7ff4100703b0 2024-01-21T11:12:49.284 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.250+0000 7ff40ffff700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7ff3f402fd50 0x7ff3f4032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff3fc0096a0 tx=0x7ff3fc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:49.284 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.254+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff404010360 con 0x7ff4100703b0 2024-01-21T11:12:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.406+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff410106b40 con 0x7ff3f402fd50 2024-01-21T11:12:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.407+0000 7ff40d7fa700 1 -- 172.21.15.89:0/3207589274 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff410106b40 con 0x7ff3f402fd50 2024-01-21T11:12:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6RXUIL8Xg+b0/51yl7ZAP3B9qqPCEfNMYVGtf8CQT29EmHL8rYqt/z4gQo/jgGPVNj7X4BVfQu7zL6RcCaB1204imTZXTnwzUYNixZPD2iB4fHQ21RDyRfZACoHztyo/naOP8mm6MZHwrXgEjjAScnbDpWGou5h46B3N1YLyFlAqHLYx3cQ8rlu4xgpQGDl8+5EzICawC1KyMsuDLawaFAh5TFQnOCVAitSJgZ8c3ZGnNBtAEQ0HmDa5f1BWh8+ZOr7jJa/rxdhrQ7LBOFNUVvMZCFhmkQrO8fu6gjZUmzlbia2fYojH7FwqELPZNY6qVQCyT1Pc/ELn0CzdfIJKb/Ku9FfDraOdONjLrmwlgo5Y1+bGH13X5oBQwvurn5nevo/Z2uPJ56hn+ygPShA0cMTNURjStOWS/ra6yd6AMtKHZCd/BHtj8Pc6atfpiLYDMkiwdTAE+kAGI+ISmDsekTqjG1lCoMtmiMCNUHNhr2sicaCR16asDfwtdJXP/8h8= ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:12:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7ff3f402fd50 msgr2=0x7ff3f4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7ff3f402fd50 0x7ff3f4032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff3fc0096a0 tx=0x7ff3fc006b40).stop 2024-01-21T11:12:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 msgr2=0x7ff41010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff41010cc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff40400b190 tx=0x7ff404006ee0).stop 2024-01-21T11:12:49.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 shutdown_connections 2024-01-21T11:12:49.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7ff3f402fd50 0x7ff3f4032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:49.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 --2- 172.21.15.89:0/3207589274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4100703b0 0x7ff41010cc10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:49.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 >> 172.21.15.89:0/3207589274 conn(0x7ff4100f63c0 msgr2=0x7ff4100f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:49.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 shutdown_connections 2024-01-21T11:12:49.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:49.409+0000 7ff416ec2700 1 -- 172.21.15.89:0/3207589274 wait complete. 2024-01-21T11:12:50.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:49.408884+0000 mgr.smithi089.ufybqw (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:51.059 INFO:teuthology.orchestra.run.smithi089.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T11:12:51.059 INFO:teuthology.orchestra.run.smithi089.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T11:12:51.060 INFO:teuthology.orchestra.run.smithi089.stderr:Adding host smithi089... 2024-01-21T11:12:51.060 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi089 172.21.15.89 2024-01-21T11:12:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.188+0000 7f51c34df700 1 Processor -- start 2024-01-21T11:12:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.188+0000 7f51c34df700 1 -- start start 2024-01-21T11:12:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.188+0000 7f51c34df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.188+0000 7f51c34df700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51bc0fc730 con 0x7f51bc0703d0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51c127b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51c127b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51280/0 (socket says 172.21.15.89:51280) 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51c127b700 1 -- 172.21.15.89:0/2016905300 learned_addr learned my addr 172.21.15.89:0/2016905300 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51c127b700 1 -- 172.21.15.89:0/2016905300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51bc101230 con 0x7f51bc0703d0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51c127b700 1 --2- 172.21.15.89:0/2016905300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc100d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f51ac00b5f0 tx=0x7f51ac006ee0).ready entity=mon.0 client_cookie=8103b8fbb2be7b14 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51abfff700 1 -- 172.21.15.89:0/2016905300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ac00d600 con 0x7f51bc0703d0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51abfff700 1 -- 172.21.15.89:0/2016905300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f51ac00dbd0 con 0x7f51bc0703d0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.189+0000 7f51abfff700 1 -- 172.21.15.89:0/2016905300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ac011c00 con 0x7f51bc0703d0 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 -- 172.21.15.89:0/2016905300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 msgr2=0x7f51bc100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 --2- 172.21.15.89:0/2016905300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc100d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f51ac00b5f0 tx=0x7f51ac006ee0).stop 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 -- 172.21.15.89:0/2016905300 shutdown_connections 2024-01-21T11:12:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 --2- 172.21.15.89:0/2016905300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc100d40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:52.193 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 -- 172.21.15.89:0/2016905300 >> 172.21.15.89:0/2016905300 conn(0x7f51bc0f63c0 msgr2=0x7f51bc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:52.193 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 -- 172.21.15.89:0/2016905300 shutdown_connections 2024-01-21T11:12:52.193 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.190+0000 7f51c34df700 1 -- 172.21.15.89:0/2016905300 wait complete. 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c34df700 1 Processor -- start 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c34df700 1 -- start start 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c34df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c34df700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51bc10d0a0 con 0x7f51bc0703d0 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c127b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c127b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51292/0 (socket says 172.21.15.89:51292) 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c127b700 1 -- 172.21.15.89:0/360525988 learned_addr learned my addr 172.21.15.89:0/360525988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.191+0000 7f51c127b700 1 -- 172.21.15.89:0/360525988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ac00b200 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.192+0000 7f51c127b700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc10cbb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f51ac00b190 tx=0x7f51ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.192+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ac01b410 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.192+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f51ac01b9e0 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.192+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ac01a880 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.192+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51bc10d2a0 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.192+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51bc10d6b0 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.193+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f51ac00f070 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.193+0000 7f51a9ffb700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f51a402fd50 0x7f51a4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.193+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f51ac041290 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.193+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51bc045ad0 con 0x7f51bc0703d0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.193+0000 7f51c0a7a700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f51a402fd50 0x7f51a4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:52.196 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.193+0000 7f51c0a7a700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f51a402fd50 0x7f51a4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f51b80096a0 tx=0x7f51b8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:52.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.197+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51ac016080 con 0x7f51bc0703d0 2024-01-21T11:12:52.368 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.367+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}) v1 -- 0x7f51bc106b30 con 0x7f51a402fd50 2024-01-21T11:12:52.959 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Added host 'smithi089' 2024-01-21T11:12:52.959 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.957+0000 7f51a9ffb700 1 -- 172.21.15.89:0/360525988 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f51bc106b30 con 0x7f51a402fd50 2024-01-21T11:12:52.961 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f51a402fd50 msgr2=0x7f51a4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f51a402fd50 0x7f51a4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f51b80096a0 tx=0x7f51b8006b40).stop 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 msgr2=0x7f51bc10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc10cbb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f51ac00b190 tx=0x7f51ac006ee0).stop 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 shutdown_connections 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f51a402fd50 0x7f51a4032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 --2- 172.21.15.89:0/360525988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51bc0703d0 0x7f51bc10cbb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 >> 172.21.15.89:0/360525988 conn(0x7f51bc0f63c0 msgr2=0x7f51bc0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 shutdown_connections 2024-01-21T11:12:52.962 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:52.960+0000 7f51c34df700 1 -- 172.21.15.89:0/360525988 wait complete. 2024-01-21T11:12:53.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:52.370047+0000 mgr.smithi089.ufybqw (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:53.264 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying mon service with default placement... 2024-01-21T11:12:53.264 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T11:12:53.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3585c71700 1 Processor -- start 2024-01-21T11:12:53.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3585c71700 1 -- start start 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3585c71700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f35800fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3585c71700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35800fec60 con 0x7f35800703f0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3584c6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f35800fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3584c6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f35800fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51300/0 (socket says 172.21.15.89:51300) 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.602+0000 7f3584c6f700 1 -- 172.21.15.89:0/2473192273 learned_addr learned my addr 172.21.15.89:0/2473192273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3584c6f700 1 -- 172.21.15.89:0/2473192273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800feda0 con 0x7f35800703f0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3584c6f700 1 --2- 172.21.15.89:0/2473192273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f35800fe770 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f357400b5f0 tx=0x7f3574006ee0).ready entity=mon.0 client_cookie=f3b232972169b0c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f357f7fe700 1 -- 172.21.15.89:0/2473192273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357400d600 con 0x7f35800703f0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f357f7fe700 1 -- 172.21.15.89:0/2473192273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f357400dbd0 con 0x7f35800703f0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f357f7fe700 1 -- 172.21.15.89:0/2473192273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3574011c00 con 0x7f35800703f0 2024-01-21T11:12:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3585c71700 1 -- 172.21.15.89:0/2473192273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 msgr2=0x7f35800fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3585c71700 1 --2- 172.21.15.89:0/2473192273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f35800fe770 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f357400b5f0 tx=0x7f3574006ee0).stop 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3585c71700 1 -- 172.21.15.89:0/2473192273 shutdown_connections 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3585c71700 1 --2- 172.21.15.89:0/2473192273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f35800fe770 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3585c71700 1 -- 172.21.15.89:0/2473192273 >> 172.21.15.89:0/2473192273 conn(0x7f35800f6210 msgr2=0x7f35800f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.603+0000 7f3585c71700 1 -- 172.21.15.89:0/2473192273 shutdown_connections 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.604+0000 7f3585c71700 1 -- 172.21.15.89:0/2473192273 wait complete. 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.604+0000 7f3585c71700 1 Processor -- start 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.604+0000 7f3585c71700 1 -- start start 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.604+0000 7f3585c71700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f358010a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.604+0000 7f3585c71700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358010ad70 con 0x7f35800703f0 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.604+0000 7f3584c6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f358010a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f3584c6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f358010a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51308/0 (socket says 172.21.15.89:51308) 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f3584c6f700 1 -- 172.21.15.89:0/1673364211 learned_addr learned my addr 172.21.15.89:0/1673364211 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:53.609 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f3584c6f700 1 -- 172.21.15.89:0/1673364211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357400b200 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f3584c6f700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f358010a880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f357400e040 tx=0x7f3574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357400d9f0 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f357401a9f0 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357401b8c0 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358010af70 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.605+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358010b380 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.606+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f357400f070 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.606+0000 7f357d7fa700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f356402fdb0 0x7f3564032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.606+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3574041300 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.606+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3580045ad0 con 0x7f35800703f0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.606+0000 7f357ffff700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f356402fdb0 0x7f3564032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.607+0000 7f357ffff700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f356402fdb0 0x7f3564032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f356c0096a0 tx=0x7f356c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:53.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.610+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3574016070 con 0x7f35800703f0 2024-01-21T11:12:53.787 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.786+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f3580104750 con 0x7f356402fdb0 2024-01-21T11:12:53.791 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.790+0000 7f357d7fa700 1 -- 172.21.15.89:0/1673364211 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3580104750 con 0x7f356402fdb0 2024-01-21T11:12:53.791 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f356402fdb0 msgr2=0x7f3564032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f356402fdb0 0x7f3564032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f356c0096a0 tx=0x7f356c006b40).stop 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 msgr2=0x7f358010a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f358010a880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f357400e040 tx=0x7f3574006ee0).stop 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 shutdown_connections 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f356402fdb0 0x7f3564032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 --2- 172.21.15.89:0/1673364211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800703f0 0x7f358010a880 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 >> 172.21.15.89:0/1673364211 conn(0x7f35800f6210 msgr2=0x7f35800f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:53.795 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 shutdown_connections 2024-01-21T11:12:53.795 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:53.792+0000 7f3585c71700 1 -- 172.21.15.89:0/1673364211 wait complete. 2024-01-21T11:12:54.064 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:52.958463+0000 mon.smithi089 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:54.064 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:12:52.958754+0000 mgr.smithi089.ufybqw (mgr.14114) 9 : cephadm [INF] Added host smithi089 2024-01-21T11:12:54.064 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:52.958976+0000 mon.smithi089 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:12:54.064 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:53.788884+0000 mgr.smithi089.ufybqw (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:54.064 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:12:53.789593+0000 mgr.smithi089.ufybqw (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T11:12:54.064 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:53.791914+0000 mon.smithi089 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying mgr service with default placement... 2024-01-21T11:12:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T11:12:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda15fb7700 1 Processor -- start 2024-01-21T11:12:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda15fb7700 1 -- start start 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda15fb7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda100fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda15fb7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda100fdca0 con 0x7fda100fd390 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda0f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda100fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda0f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda100fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51312/0 (socket says 172.21.15.89:51312) 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda0f7fe700 1 -- 172.21.15.89:0/2644587704 learned_addr learned my addr 172.21.15.89:0/2644587704 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda0f7fe700 1 -- 172.21.15.89:0/2644587704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda100fdde0 con 0x7fda100fd390 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.470+0000 7fda0f7fe700 1 --2- 172.21.15.89:0/2644587704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda100fd7b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fda0000b5f0 tx=0x7fda00006ee0).ready entity=mon.0 client_cookie=fc69408ed279ab62 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda0e7fc700 1 -- 172.21.15.89:0/2644587704 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda0000d600 con 0x7fda100fd390 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda0e7fc700 1 -- 172.21.15.89:0/2644587704 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fda0000dbd0 con 0x7fda100fd390 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda0e7fc700 1 -- 172.21.15.89:0/2644587704 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda00011c00 con 0x7fda100fd390 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 -- 172.21.15.89:0/2644587704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 msgr2=0x7fda100fd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 --2- 172.21.15.89:0/2644587704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda100fd7b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fda0000b5f0 tx=0x7fda00006ee0).stop 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 -- 172.21.15.89:0/2644587704 shutdown_connections 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 --2- 172.21.15.89:0/2644587704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda100fd7b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 -- 172.21.15.89:0/2644587704 >> 172.21.15.89:0/2644587704 conn(0x7fda100f5b10 msgr2=0x7fda100f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 -- 172.21.15.89:0/2644587704 shutdown_connections 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.471+0000 7fda15fb7700 1 -- 172.21.15.89:0/2644587704 wait complete. 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda15fb7700 1 Processor -- start 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda15fb7700 1 -- start start 2024-01-21T11:12:54.475 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda15fb7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda1010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:54.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda15fb7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1010af00 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda0f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda1010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda0f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda1010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51314/0 (socket says 172.21.15.89:51314) 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda0f7fe700 1 -- 172.21.15.89:0/1332108223 learned_addr learned my addr 172.21.15.89:0/1332108223 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.472+0000 7fda0f7fe700 1 -- 172.21.15.89:0/1332108223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0000b200 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.473+0000 7fda0f7fe700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda1010aa10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fda00002f40 tx=0x7fda00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.473+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda0001b410 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.473+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fda0001b9e0 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.473+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1010b100 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.473+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda0001a8b0 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.473+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1010b510 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.474+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fda0000f070 con 0x7fda100fd390 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.474+0000 7fd9f7fff700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fd9fc02fdb0 0x7fd9fc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:54.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.474+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fda00041100 con 0x7fda100fd390 2024-01-21T11:12:54.478 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.474+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda10045ad0 con 0x7fda100fd390 2024-01-21T11:12:54.478 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.474+0000 7fda0effd700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fd9fc02fdb0 0x7fd9fc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:54.478 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.474+0000 7fda0effd700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fd9fc02fdb0 0x7fd9fc032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fda040096a0 tx=0x7fda04006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:54.479 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.478+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda00016070 con 0x7fda100fd390 2024-01-21T11:12:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.652+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fda1010bdf0 con 0x7fd9fc02fdb0 2024-01-21T11:12:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.655+0000 7fd9f7fff700 1 -- 172.21.15.89:0/1332108223 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fda1010bdf0 con 0x7fd9fc02fdb0 2024-01-21T11:12:54.657 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T11:12:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fd9fc02fdb0 msgr2=0x7fd9fc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fd9fc02fdb0 0x7fd9fc032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fda040096a0 tx=0x7fda04006b40).stop 2024-01-21T11:12:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 msgr2=0x7fda1010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda1010aa10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fda00002f40 tx=0x7fda00006ee0).stop 2024-01-21T11:12:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 shutdown_connections 2024-01-21T11:12:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fd9fc02fdb0 0x7fd9fc032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 --2- 172.21.15.89:0/1332108223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda100fd390 0x7fda1010aa10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.657+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 >> 172.21.15.89:0/1332108223 conn(0x7fda100f5b10 msgr2=0x7fda100ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.658+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 shutdown_connections 2024-01-21T11:12:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:54.658+0000 7fda15fb7700 1 -- 172.21.15.89:0/1332108223 wait complete. 2024-01-21T11:12:54.987 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying crash service with default placement... 2024-01-21T11:12:54.987 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T11:12:55.322 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f0948e2b700 1 Processor -- start 2024-01-21T11:12:55.322 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f0948e2b700 1 -- start start 2024-01-21T11:12:55.322 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f0948e2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f09440fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:55.322 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f0948e2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440fedd0 con 0x7f0944070430 2024-01-21T11:12:55.322 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f094259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f09440fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f094259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f09440fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51330/0 (socket says 172.21.15.89:51330) 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.319+0000 7f094259c700 1 -- 172.21.15.89:0/3340266697 learned_addr learned my addr 172.21.15.89:0/3340266697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.320+0000 7f094259c700 1 -- 172.21.15.89:0/3340266697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440fef10 con 0x7f0944070430 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.320+0000 7f094259c700 1 --2- 172.21.15.89:0/3340266697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f09440fe8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f093400b5f0 tx=0x7f0934006ee0).ready entity=mon.0 client_cookie=e54442fc34060e26 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.320+0000 7f094159a700 1 -- 172.21.15.89:0/3340266697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093400d600 con 0x7f0944070430 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.320+0000 7f094159a700 1 -- 172.21.15.89:0/3340266697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f093400dbd0 con 0x7f0944070430 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.320+0000 7f094159a700 1 -- 172.21.15.89:0/3340266697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0934011c00 con 0x7f0944070430 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- 172.21.15.89:0/3340266697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 msgr2=0x7f09440fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 --2- 172.21.15.89:0/3340266697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f09440fe8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f093400b5f0 tx=0x7f0934006ee0).stop 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- 172.21.15.89:0/3340266697 shutdown_connections 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 --2- 172.21.15.89:0/3340266697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f09440fe8e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- 172.21.15.89:0/3340266697 >> 172.21.15.89:0/3340266697 conn(0x7f09440f63c0 msgr2=0x7f09440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- 172.21.15.89:0/3340266697 shutdown_connections 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- 172.21.15.89:0/3340266697 wait complete. 2024-01-21T11:12:55.323 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 Processor -- start 2024-01-21T11:12:55.324 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- start start 2024-01-21T11:12:55.324 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f094410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:55.324 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f0948e2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094410aef0 con 0x7f0944070430 2024-01-21T11:12:55.325 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f094259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f094410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.321+0000 7f094259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f094410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51342/0 (socket says 172.21.15.89:51342) 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f094259c700 1 -- 172.21.15.89:0/3252980689 learned_addr learned my addr 172.21.15.89:0/3252980689 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f094259c700 1 -- 172.21.15.89:0/3252980689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093400b200 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f094259c700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f094410aa00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f093400b190 tx=0x7f0934006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093401b410 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f093401b9e0 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094410b0f0 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093401a880 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.322+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094410b500 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.323+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f093400f070 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.323+0000 7f0932ffd700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f092802fd50 0x7f0928032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.323+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f09340412d0 con 0x7f0944070430 2024-01-21T11:12:55.326 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.323+0000 7f0941d9b700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f092802fd50 0x7f0928032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:55.327 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.323+0000 7f0941d9b700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f092802fd50 0x7f0928032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f09380096a0 tx=0x7f0938006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:55.327 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.323+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944059100 con 0x7f0944070430 2024-01-21T11:12:55.328 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.328+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0934016070 con 0x7f0944070430 2024-01-21T11:12:55.502 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.501+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f09441048f0 con 0x7f092802fd50 2024-01-21T11:12:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.505+0000 7f0932ffd700 1 -- 172.21.15.89:0/3252980689 <== mgr.14114 v2:172.21.15.89:6800/4200893972 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f09441048f0 con 0x7f092802fd50 2024-01-21T11:12:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f092802fd50 msgr2=0x7f0928032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f092802fd50 0x7f0928032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f09380096a0 tx=0x7f0938006b40).stop 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 msgr2=0x7f094410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f094410aa00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f093400b190 tx=0x7f0934006ee0).stop 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 shutdown_connections 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f092802fd50 0x7f0928032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 --2- 172.21.15.89:0/3252980689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944070430 0x7f094410aa00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:55.509 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 >> 172.21.15.89:0/3252980689 conn(0x7f09440f63c0 msgr2=0x7f09440f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:55.509 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 shutdown_connections 2024-01-21T11:12:55.509 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:55.506+0000 7f0948e2b700 1 -- 172.21.15.89:0/3252980689 wait complete. 2024-01-21T11:12:55.797 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:54.654794+0000 mgr.smithi089.ufybqw (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:55.797 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:12:54.655300+0000 mgr.smithi089.ufybqw (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T11:12:55.797 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:54.657123+0000 mon.smithi089 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:55.797 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:55.506227+0000 mon.smithi089 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.89:0/60843299' entity='mgr.smithi089.ufybqw' 2024-01-21T11:12:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:Enabling mgr prometheus module... 2024-01-21T11:12:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T11:12:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd17024700 1 Processor -- start 2024-01-21T11:12:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd17024700 1 -- start start 2024-01-21T11:12:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd17024700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd100fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd17024700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd100fedd0 con 0x7fcd10070430 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd14dc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd100fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd14dc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd100fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51348/0 (socket says 172.21.15.89:51348) 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.194+0000 7fcd14dc0700 1 -- 172.21.15.89:0/3337306 learned_addr learned my addr 172.21.15.89:0/3337306 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.195+0000 7fcd14dc0700 1 -- 172.21.15.89:0/3337306 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd100fef10 con 0x7fcd10070430 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.195+0000 7fcd14dc0700 1 --2- 172.21.15.89:0/3337306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd100fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fcd0400b5f0 tx=0x7fcd04006ee0).ready entity=mon.0 client_cookie=b233b0ecf67d2f7d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.195+0000 7fcd0f7fe700 1 -- 172.21.15.89:0/3337306 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd0400d600 con 0x7fcd10070430 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.195+0000 7fcd0f7fe700 1 -- 172.21.15.89:0/3337306 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcd0400dbd0 con 0x7fcd10070430 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.195+0000 7fcd0f7fe700 1 -- 172.21.15.89:0/3337306 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd04011c00 con 0x7fcd10070430 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 -- 172.21.15.89:0/3337306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 msgr2=0x7fcd100fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 --2- 172.21.15.89:0/3337306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd100fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fcd0400b5f0 tx=0x7fcd04006ee0).stop 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 -- 172.21.15.89:0/3337306 shutdown_connections 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 --2- 172.21.15.89:0/3337306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd100fe8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 -- 172.21.15.89:0/3337306 >> 172.21.15.89:0/3337306 conn(0x7fcd100f63c0 msgr2=0x7fcd100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 -- 172.21.15.89:0/3337306 shutdown_connections 2024-01-21T11:12:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 -- 172.21.15.89:0/3337306 wait complete. 2024-01-21T11:12:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 Processor -- start 2024-01-21T11:12:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.196+0000 7fcd17024700 1 -- start start 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd17024700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd1010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd17024700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1010aef0 con 0x7fcd10070430 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd14dc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd1010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd14dc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd1010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51360/0 (socket says 172.21.15.89:51360) 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd14dc0700 1 -- 172.21.15.89:0/2952673018 learned_addr learned my addr 172.21.15.89:0/2952673018 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd14dc0700 1 -- 172.21.15.89:0/2952673018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0400b200 con 0x7fcd10070430 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd14dc0700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd1010aa00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcd0400b190 tx=0x7fcd04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd0401b410 con 0x7fcd10070430 2024-01-21T11:12:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcd0401b9e0 con 0x7fcd10070430 2024-01-21T11:12:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1010b0f0 con 0x7fcd10070430 2024-01-21T11:12:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.197+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd0401a880 con 0x7fcd10070430 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.198+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1010b500 con 0x7fcd10070430 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.198+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcd0400f070 con 0x7fcd10070430 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.198+0000 7fcd0d7fa700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fccf402fdb0 0x7fccf4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.198+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fcd04041200 con 0x7fcd10070430 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.198+0000 7fcd0ffff700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fccf402fdb0 0x7fccf4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.199+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd10059100 con 0x7fcd10070430 2024-01-21T11:12:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.199+0000 7fcd0ffff700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fccf402fdb0 0x7fccf4032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fccfc0096a0 tx=0x7fccfc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:56.203 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.203+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd04016070 con 0x7fcd10070430 2024-01-21T11:12:56.389 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.388+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fcd10045ad0 con 0x7fcd10070430 2024-01-21T11:12:56.660 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.659+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fcd040257e0 con 0x7fcd10070430 2024-01-21T11:12:56.661 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.660+0000 7fcd0d7fa700 1 -- 172.21.15.89:0/2952673018 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7fcd04006a70 con 0x7fcd10070430 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fccf402fdb0 msgr2=0x7fccf4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fccf402fdb0 0x7fccf4032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fccfc0096a0 tx=0x7fccfc006b40).stop 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 msgr2=0x7fcd1010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd1010aa00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcd0400b190 tx=0x7fcd04006ee0).stop 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 shutdown_connections 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7fccf402fdb0 0x7fccf4032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 --2- 172.21.15.89:0/2952673018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd10070430 0x7fcd1010aa00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 >> 172.21.15.89:0/2952673018 conn(0x7fcd100f63c0 msgr2=0x7fcd100f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 shutdown_connections 2024-01-21T11:12:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:56.662+0000 7fcd17024700 1 -- 172.21.15.89:0/2952673018 wait complete. 2024-01-21T11:12:56.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:55.503341+0000 mgr.smithi089.ufybqw (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:12:56.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:12:55.504015+0000 mgr.smithi089.ufybqw (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T11:12:56.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:56.389983+0000 mon.smithi089 (mon.0) 58 : audit [INF] from='client.? 172.21.15.89:0/2952673018' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T11:12:57.003 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying prometheus service with default placement... 2024-01-21T11:12:57.003 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T11:12:57.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40f5a5e700 1 Processor -- start 2024-01-21T11:12:57.413 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40f5a5e700 1 -- start start 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40f5a5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f00fae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40f5a5e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f00fb320 con 0x7f40f00faa10 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f00fae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f00fae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51394/0 (socket says 172.21.15.89:51394) 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.410+0000 7f40eeffd700 1 -- 172.21.15.89:0/3972006023 learned_addr learned my addr 172.21.15.89:0/3972006023 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40eeffd700 1 -- 172.21.15.89:0/3972006023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f00fb460 con 0x7f40f00faa10 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40eeffd700 1 --2- 172.21.15.89:0/3972006023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f00fae30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f40e000b5f0 tx=0x7f40e0006ee0).ready entity=mon.0 client_cookie=bc1860b8588d2cf6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40edffb700 1 -- 172.21.15.89:0/3972006023 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e000d600 con 0x7f40f00faa10 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40edffb700 1 -- 172.21.15.89:0/3972006023 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40e000dbd0 con 0x7f40f00faa10 2024-01-21T11:12:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40edffb700 1 -- 172.21.15.89:0/3972006023 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e0011c00 con 0x7f40f00faa10 2024-01-21T11:12:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40f5a5e700 1 -- 172.21.15.89:0/3972006023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 msgr2=0x7f40f00fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:12:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/3972006023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f00fae30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f40e000b5f0 tx=0x7f40e0006ee0).stop 2024-01-21T11:12:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40f5a5e700 1 -- 172.21.15.89:0/3972006023 shutdown_connections 2024-01-21T11:12:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/3972006023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f00fae30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:12:57.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.411+0000 7f40f5a5e700 1 -- 172.21.15.89:0/3972006023 >> 172.21.15.89:0/3972006023 conn(0x7f40f00f6250 msgr2=0x7f40f00f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:12:57.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.412+0000 7f40f5a5e700 1 -- 172.21.15.89:0/3972006023 shutdown_connections 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.412+0000 7f40f5a5e700 1 -- 172.21.15.89:0/3972006023 wait complete. 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.412+0000 7f40f5a5e700 1 Processor -- start 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.412+0000 7f40f5a5e700 1 -- start start 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.412+0000 7f40f5a5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f01097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.412+0000 7f40f5a5e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0109cd0 con 0x7f40f00faa10 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f01097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f01097e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51410/0 (socket says 172.21.15.89:51410) 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40eeffd700 1 -- 172.21.15.89:0/1459950448 learned_addr learned my addr 172.21.15.89:0/1459950448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40eeffd700 1 -- 172.21.15.89:0/1459950448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e000b200 con 0x7f40f00faa10 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40eeffd700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f01097e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f40e000e040 tx=0x7f40e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e001b410 con 0x7f40f00faa10 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f0109ed0 con 0x7f40f00faa10 2024-01-21T11:12:57.418 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40e001b9e0 con 0x7f40f00faa10 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e001a770 con 0x7f40f00faa10 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.413+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f010a2e0 con 0x7f40f00faa10 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.414+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f40e000f070 con 0x7f40f00faa10 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.414+0000 7f40d77fe700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.414+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f40e00410e0 con 0x7f40f00faa10 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.414+0000 7f40ee7fc700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 msgr2=0x7f40dc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4200893972 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.414+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.415+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f0059100 con 0x7f40f00faa10 2024-01-21T11:12:57.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.419+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40e0016070 con 0x7f40f00faa10 2024-01-21T11:12:57.603 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.602+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 --> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f40f0103630 con 0x7f40dc02fdb0 2024-01-21T11:12:57.615 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.615+0000 7f40ee7fc700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 msgr2=0x7f40dc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4200893972 2024-01-21T11:12:57.616 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:57.615+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:12:57.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:57 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:12:56.662035+0000 mon.smithi089 (mon.0) 59 : audit [INF] from='client.? 172.21.15.89:0/2952673018' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T11:12:57.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:12:57 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:12:56.662100+0000 mon.smithi089 (mon.0) 60 : cluster [DBG] mgrmap e8: smithi089.ufybqw(active, since 12s) 2024-01-21T11:12:58.017 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:58.016+0000 7f40ee7fc700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 msgr2=0x7f40dc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4200893972 2024-01-21T11:12:58.017 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:58.016+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T11:12:58.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:58.817+0000 7f40ee7fc700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 msgr2=0x7f40dc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4200893972 2024-01-21T11:12:58.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:12:58.817+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T11:13:00.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:00.419+0000 7f40ee7fc700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 msgr2=0x7f40dc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/4200893972 2024-01-21T11:13:00.419 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:00.419+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T11:13:01.628 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:01.628+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f40e0024790 con 0x7f40f00faa10 2024-01-21T11:13:01.629 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:01.628+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 msgr2=0x7f40dc032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:13:01.629 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:01.628+0000 7f40d77fe700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:13:01.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:01.577231+0000 mon.smithi089 (mon.0) 61 : cluster [INF] Active manager daemon smithi089.ufybqw restarted 2024-01-21T11:13:01.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:01.577643+0000 mon.smithi089 (mon.0) 62 : cluster [INF] Activating manager daemon smithi089.ufybqw 2024-01-21T11:13:01.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:01.579491+0000 mon.smithi089 (mon.0) 63 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T11:13:02.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.635+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f40e0042f60 con 0x7f40f00faa10 2024-01-21T11:13:02.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.635+0000 7f40d77fe700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f40dc033c20 0x7f40dc036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:02.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.635+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 --> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f40f0103630 con 0x7f40dc033c20 2024-01-21T11:13:02.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.636+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f40dc033c20 0x7f40dc036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:02.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.637+0000 7f40ee7fc700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f40dc033c20 0x7f40dc036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f40e40036d0 tx=0x7f40e4006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:02.641 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.642+0000 7f40d77fe700 1 -- 172.21.15.89:0/1459950448 <== mgr.14142 v2:172.21.15.89:6800/578651808 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f40f0103630 con 0x7f40dc033c20 2024-01-21T11:13:02.642 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f40dc033c20 msgr2=0x7f40dc036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f40dc033c20 0x7f40dc036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f40e40036d0 tx=0x7f40e4006cb0).stop 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 msgr2=0x7f40f01097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f01097e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f40e000e040 tx=0x7f40e0006ee0).stop 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 shutdown_connections 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f40dc033c20 0x7f40dc036010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f00faa10 0x7f40f01097e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 --2- 172.21.15.89:0/1459950448 >> [v2:172.21.15.89:6800/4200893972,v1:172.21.15.89:6801/4200893972] conn(0x7f40dc02fdb0 0x7f40dc032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 >> 172.21.15.89:0/1459950448 conn(0x7f40f00f6250 msgr2=0x7f40f00f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 shutdown_connections 2024-01-21T11:13:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:02.644+0000 7f40f5a5e700 1 -- 172.21.15.89:0/1459950448 wait complete. 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:01.630115+0000 mon.smithi089 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi089.ufybqw(active, starting, since 0.0525569s) 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.632282+0000 mon.smithi089 (mon.0) 65 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.632493+0000 mon.smithi089 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mgr metadata", "who": "smithi089.ufybqw", "id": "smithi089.ufybqw"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.632654+0000 mon.smithi089 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.632786+0000 mon.smithi089 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.632921+0000 mon.smithi089 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:01.646934+0000 mon.smithi089 (mon.0) 70 : cluster [INF] Manager daemon smithi089.ufybqw is now available 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.742811+0000 mon.smithi089 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.744208+0000 mon.smithi089 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.759652+0000 mon.smithi089 (mon.0) 73 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:13:02.922 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:01.767913+0000 mon.smithi089 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:13:02.935 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying grafana service with default placement... 2024-01-21T11:13:02.935 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T11:13:03.284 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.284+0000 7f4342993700 1 Processor -- start 2024-01-21T11:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.284+0000 7f4342993700 1 -- start start 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.284+0000 7f4342993700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.284+0000 7f4342993700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0fec80 con 0x7f433c0703f0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f4341991700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f4341991700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60056/0 (socket says 172.21.15.89:60056) 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f4341991700 1 -- 172.21.15.89:0/1960310280 learned_addr learned my addr 172.21.15.89:0/1960310280 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f4341991700 1 -- 172.21.15.89:0/1960310280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0fedc0 con 0x7f433c0703f0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f4341991700 1 --2- 172.21.15.89:0/1960310280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fe790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f433800b5f0 tx=0x7f4338006ee0).ready entity=mon.0 client_cookie=918e7ba57c77cf30 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f434098f700 1 -- 172.21.15.89:0/1960310280 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f433800d600 con 0x7f433c0703f0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f434098f700 1 -- 172.21.15.89:0/1960310280 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f433800dbd0 con 0x7f433c0703f0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.285+0000 7f434098f700 1 -- 172.21.15.89:0/1960310280 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4338011c00 con 0x7f433c0703f0 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 -- 172.21.15.89:0/1960310280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 msgr2=0x7f433c0fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 --2- 172.21.15.89:0/1960310280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fe790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f433800b5f0 tx=0x7f4338006ee0).stop 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 -- 172.21.15.89:0/1960310280 shutdown_connections 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 --2- 172.21.15.89:0/1960310280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fe790 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:03.287 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 -- 172.21.15.89:0/1960310280 >> 172.21.15.89:0/1960310280 conn(0x7f433c0f6210 msgr2=0x7f433c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 -- 172.21.15.89:0/1960310280 shutdown_connections 2024-01-21T11:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 -- 172.21.15.89:0/1960310280 wait complete. 2024-01-21T11:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 Processor -- start 2024-01-21T11:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.286+0000 7f4342993700 1 -- start start 2024-01-21T11:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4342993700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4342993700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0fa690 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4341991700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4341991700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60058/0 (socket says 172.21.15.89:60058) 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4341991700 1 -- 172.21.15.89:0/976718166 learned_addr learned my addr 172.21.15.89:0/976718166 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4341991700 1 -- 172.21.15.89:0/976718166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433800b200 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4341991700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fc210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f433800e040 tx=0x7f4338006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f433800d9f0 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f433801a9f0 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f433801b8c0 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c0fa830 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.287+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c0fac40 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.288+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f433800f070 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.288+0000 7f432a7fc700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f432402fd70 0x7f4324032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.288+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4338041290 con 0x7f433c0703f0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.288+0000 7f4341190700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f432402fd70 0x7f4324032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.288+0000 7f4341190700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f432402fd70 0x7f4324032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f432c0096a0 tx=0x7f432c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.289+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c059100 con 0x7f433c0703f0 2024-01-21T11:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.293+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43380224f0 con 0x7f433c0703f0 2024-01-21T11:13:03.469 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.470+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 --> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f433c104730 con 0x7f432402fd70 2024-01-21T11:13:03.473 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.473+0000 7f432a7fc700 1 -- 172.21.15.89:0/976718166 <== mgr.14142 v2:172.21.15.89:6800/578651808 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f433c104730 con 0x7f432402fd70 2024-01-21T11:13:03.473 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f432402fd70 msgr2=0x7f4324032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f432402fd70 0x7f4324032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f432c0096a0 tx=0x7f432c006b40).stop 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 msgr2=0x7f433c0fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fc210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f433800e040 tx=0x7f4338006ee0).stop 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 shutdown_connections 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f432402fd70 0x7f4324032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 --2- 172.21.15.89:0/976718166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f433c0703f0 0x7f433c0fc210 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.475+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 >> 172.21.15.89:0/976718166 conn(0x7f433c0f6210 msgr2=0x7f433c0f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:03.476 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.476+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 shutdown_connections 2024-01-21T11:13:03.477 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:03.476+0000 7f4342993700 1 -- 172.21.15.89:0/976718166 wait complete. 2024-01-21T11:13:03.762 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:03 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:02.638093+0000 mon.smithi089 (mon.0) 75 : cluster [DBG] mgrmap e10: smithi089.ufybqw(active, since 1.06053s) 2024-01-21T11:13:03.762 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:03 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:02.639511+0000 mgr.smithi089.ufybqw (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T11:13:03.762 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:03 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:02.642332+0000 mon.smithi089 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:03.762 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:03 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:03.474022+0000 mon.smithi089 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:03.836 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying node-exporter service with default placement... 2024-01-21T11:13:03.836 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.168+0000 7fa3693a2700 1 Processor -- start 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.168+0000 7fa3693a2700 1 -- start start 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa3693a2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa3640fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa3693a2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640fb2e0 con 0x7fa3640fa9d0 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa363fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa3640fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa363fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa3640fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60074/0 (socket says 172.21.15.89:60074) 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa363fff700 1 -- 172.21.15.89:0/45133759 learned_addr learned my addr 172.21.15.89:0/45133759 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa363fff700 1 -- 172.21.15.89:0/45133759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3640fb420 con 0x7fa3640fa9d0 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa363fff700 1 --2- 172.21.15.89:0/45133759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa3640fadf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa35800b870 tx=0x7fa358006f70).ready entity=mon.0 client_cookie=8aa08dc63af7a6f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:04.172 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa362ffd700 1 -- 172.21.15.89:0/45133759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35800da20 con 0x7fa3640fa9d0 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa362ffd700 1 -- 172.21.15.89:0/45133759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa358010470 con 0x7fa3640fa9d0 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.169+0000 7fa362ffd700 1 -- 172.21.15.89:0/45133759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35800f5b0 con 0x7fa3640fa9d0 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.170+0000 7fa3693a2700 1 -- 172.21.15.89:0/45133759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 msgr2=0x7fa3640fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.170+0000 7fa3693a2700 1 --2- 172.21.15.89:0/45133759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa3640fadf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa35800b870 tx=0x7fa358006f70).stop 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.170+0000 7fa3693a2700 1 -- 172.21.15.89:0/45133759 shutdown_connections 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.170+0000 7fa3693a2700 1 --2- 172.21.15.89:0/45133759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa3640fadf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.170+0000 7fa3693a2700 1 -- 172.21.15.89:0/45133759 >> 172.21.15.89:0/45133759 conn(0x7fa3640f6210 msgr2=0x7fa3640f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa3693a2700 1 -- 172.21.15.89:0/45133759 shutdown_connections 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa3693a2700 1 -- 172.21.15.89:0/45133759 wait complete. 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa3693a2700 1 Processor -- start 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa3693a2700 1 -- start start 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa3693a2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa364108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa3693a2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa364108a60 con 0x7fa3640fa9d0 2024-01-21T11:13:04.173 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa363fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa364108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:04.174 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.171+0000 7fa363fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa364108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60076/0 (socket says 172.21.15.89:60076) 2024-01-21T11:13:04.174 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa363fff700 1 -- 172.21.15.89:0/1574194369 learned_addr learned my addr 172.21.15.89:0/1574194369 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa363fff700 1 -- 172.21.15.89:0/1574194369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35800b480 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa363fff700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa364108570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa358006a40 tx=0x7fa358006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35800f3d0 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa364108c60 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa35800fef0 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa358019d40 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.172+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa364109070 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.173+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa364045ad0 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.174+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fa35802b060 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.174+0000 7fa360ff9700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fa35002fcf0 0x7fa3500321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.174+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa3580443d0 con 0x7fa3640fa9d0 2024-01-21T11:13:04.175 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.174+0000 7fa3637fe700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fa35002fcf0 0x7fa3500321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:04.176 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.174+0000 7fa3637fe700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fa35002fcf0 0x7fa3500321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa3540096a0 tx=0x7fa354006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:04.177 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.178+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa35800e950 con 0x7fa3640fa9d0 2024-01-21T11:13:04.352 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.352+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 --> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3641099b0 con 0x7fa35002fcf0 2024-01-21T11:13:04.355 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.356+0000 7fa360ff9700 1 -- 172.21.15.89:0/1574194369 <== mgr.14142 v2:172.21.15.89:6800/578651808 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fa3641099b0 con 0x7fa35002fcf0 2024-01-21T11:13:04.355 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T11:13:04.358 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.357+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fa35002fcf0 msgr2=0x7fa3500321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:04.358 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.357+0000 7fa3693a2700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fa35002fcf0 0x7fa3500321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa3540096a0 tx=0x7fa354006b40).stop 2024-01-21T11:13:04.358 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.357+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 msgr2=0x7fa364108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.357+0000 7fa3693a2700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa364108570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa358006a40 tx=0x7fa358006f70).stop 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.358+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 shutdown_connections 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.358+0000 7fa3693a2700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fa35002fcf0 0x7fa3500321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.358+0000 7fa3693a2700 1 --2- 172.21.15.89:0/1574194369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3640fa9d0 0x7fa364108570 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.358+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 >> 172.21.15.89:0/1574194369 conn(0x7fa3640f6210 msgr2=0x7fa3640ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.358+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 shutdown_connections 2024-01-21T11:13:04.359 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:04.358+0000 7fa3693a2700 1 -- 172.21.15.89:0/1574194369 wait complete. 2024-01-21T11:13:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:Deploying alertmanager service with default placement... 2024-01-21T11:13:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T11:13:04.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:03.471311+0000 mgr.smithi089.ufybqw (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:04.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:03.471814+0000 mgr.smithi089.ufybqw (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T11:13:04.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:04.356153+0000 mon.smithi089 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:04.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:04.476339+0000 mon.smithi089 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi089.ufybqw(active, since 2s) 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15abfff700 1 Processor -- start 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15abfff700 1 -- start start 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15abfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15abfff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0fb2e0 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60084/0 (socket says 172.21.15.89:60084) 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15aaffd700 1 -- 172.21.15.89:0/4233329509 learned_addr learned my addr 172.21.15.89:0/4233329509 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.040+0000 7f15aaffd700 1 -- 172.21.15.89:0/4233329509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0fb420 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15aaffd700 1 --2- 172.21.15.89:0/4233329509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac0fadf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f159c00b5f0 tx=0x7f159c006ee0).ready entity=mon.0 client_cookie=41a8e95e6a3fd91e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15a9ffb700 1 -- 172.21.15.89:0/4233329509 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159c00d600 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.043 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15a9ffb700 1 -- 172.21.15.89:0/4233329509 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f159c00dbd0 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15a9ffb700 1 -- 172.21.15.89:0/4233329509 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159c011c00 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 -- 172.21.15.89:0/4233329509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 msgr2=0x7f15ac0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 --2- 172.21.15.89:0/4233329509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac0fadf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f159c00b5f0 tx=0x7f159c006ee0).stop 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 -- 172.21.15.89:0/4233329509 shutdown_connections 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 --2- 172.21.15.89:0/4233329509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac0fadf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 -- 172.21.15.89:0/4233329509 >> 172.21.15.89:0/4233329509 conn(0x7f15ac0f6210 msgr2=0x7f15ac0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 -- 172.21.15.89:0/4233329509 shutdown_connections 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.041+0000 7f15abfff700 1 -- 172.21.15.89:0/4233329509 wait complete. 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.042+0000 7f15abfff700 1 Processor -- start 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.042+0000 7f15abfff700 1 -- start start 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15abfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac10a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15abfff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac10acf0 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.045 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac10a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:05.045 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac10a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60086/0 (socket says 172.21.15.89:60086) 2024-01-21T11:13:05.045 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15aaffd700 1 -- 172.21.15.89:0/2778904559 learned_addr learned my addr 172.21.15.89:0/2778904559 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:05.046 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15aaffd700 1 -- 172.21.15.89:0/2778904559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c00b200 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.046 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15aaffd700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac10a800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f159c00b1d0 tx=0x7f159c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:05.046 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159c01b410 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.046 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f159c01b9e0 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.046 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159c01a840 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac10aef0 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.043+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac10b300 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.044+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f159c00f070 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.044+0000 7f15937fe700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f159802fe60 0x7f1598032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.044+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac045ad0 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.044+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f159c041420 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.044+0000 7f15aa7fc700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f159802fe60 0x7f1598032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:05.047 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.045+0000 7f15aa7fc700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f159802fe60 0x7f1598032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f15a00096a0 tx=0x7f15a0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:05.048 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.048+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f159c025460 con 0x7f15ac0fa9d0 2024-01-21T11:13:05.223 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.222+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 --> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f15ac10bbe0 con 0x7f159802fe60 2024-01-21T11:13:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.226+0000 7f15937fe700 1 -- 172.21.15.89:0/2778904559 <== mgr.14142 v2:172.21.15.89:6800/578651808 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f15ac10bbe0 con 0x7f159802fe60 2024-01-21T11:13:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f159802fe60 msgr2=0x7f1598032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f159802fe60 0x7f1598032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f15a00096a0 tx=0x7f15a0006b40).stop 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 msgr2=0x7f15ac10a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac10a800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f159c00b1d0 tx=0x7f159c006ee0).stop 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 shutdown_connections 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f159802fe60 0x7f1598032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 --2- 172.21.15.89:0/2778904559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15ac0fa9d0 0x7f15ac10a800 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 >> 172.21.15.89:0/2778904559 conn(0x7f15ac0f6210 msgr2=0x7f15ac070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 shutdown_connections 2024-01-21T11:13:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.228+0000 7f15abfff700 1 -- 172.21.15.89:0/2778904559 wait complete. 2024-01-21T11:13:05.577 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.913+0000 7f59ff8f1700 1 Processor -- start 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.913+0000 7f59ff8f1700 1 -- start start 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.913+0000 7f59ff8f1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.913+0000 7f59ff8f1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f80fb490 con 0x7f59f80fab80 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59fd68d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59fd68d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60100/0 (socket says 172.21.15.89:60100) 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59fd68d700 1 -- 172.21.15.89:0/731921472 learned_addr learned my addr 172.21.15.89:0/731921472 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59fd68d700 1 -- 172.21.15.89:0/731921472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f80fb5d0 con 0x7f59f80fab80 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59fd68d700 1 --2- 172.21.15.89:0/731921472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f80fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f59e800b5f0 tx=0x7f59e8006ee0).ready entity=mon.0 client_cookie=6a08d2672a393bc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:05.917 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59e7fff700 1 -- 172.21.15.89:0/731921472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e800d600 con 0x7f59f80fab80 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59e7fff700 1 -- 172.21.15.89:0/731921472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f59e800dbd0 con 0x7f59f80fab80 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.914+0000 7f59e7fff700 1 -- 172.21.15.89:0/731921472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e8011c00 con 0x7f59f80fab80 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 -- 172.21.15.89:0/731921472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 msgr2=0x7f59f80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 --2- 172.21.15.89:0/731921472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f80fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f59e800b5f0 tx=0x7f59e8006ee0).stop 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 -- 172.21.15.89:0/731921472 shutdown_connections 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 --2- 172.21.15.89:0/731921472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f80fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 -- 172.21.15.89:0/731921472 >> 172.21.15.89:0/731921472 conn(0x7f59f80f63c0 msgr2=0x7f59f80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 -- 172.21.15.89:0/731921472 shutdown_connections 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.915+0000 7f59ff8f1700 1 -- 172.21.15.89:0/731921472 wait complete. 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59ff8f1700 1 Processor -- start 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59ff8f1700 1 -- start start 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59ff8f1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59ff8f1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f810aee0 con 0x7f59f80fab80 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59fd68d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:05.918 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59fd68d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60112/0 (socket says 172.21.15.89:60112) 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59fd68d700 1 -- 172.21.15.89:0/2444676191 learned_addr learned my addr 172.21.15.89:0/2444676191 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59fd68d700 1 -- 172.21.15.89:0/2444676191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e800b200 con 0x7f59f80fab80 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59fd68d700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f810a9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f59e800b1d0 tx=0x7f59e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e801b410 con 0x7f59f80fab80 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f810b0e0 con 0x7f59f80fab80 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.916+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f59e801b9e0 con 0x7f59f80fab80 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.917+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e801a840 con 0x7f59f80fab80 2024-01-21T11:13:05.919 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.917+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f810b4f0 con 0x7f59f80fab80 2024-01-21T11:13:05.920 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.918+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f59e800f070 con 0x7f59f80fab80 2024-01-21T11:13:05.920 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.918+0000 7f59e5ffb700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f59e002fe60 0x7f59e0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:05.920 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.918+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f8045ad0 con 0x7f59f80fab80 2024-01-21T11:13:05.920 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.918+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f59e8041420 con 0x7f59f80fab80 2024-01-21T11:13:05.920 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.918+0000 7f59fce8c700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f59e002fe60 0x7f59e0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:05.920 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.918+0000 7f59fce8c700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f59e002fe60 0x7f59e0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f59f40096a0 tx=0x7f59f4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:05.922 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:05.922+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59e8025460 con 0x7f59f80fab80 2024-01-21T11:13:05.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:04.353266+0000 mgr.smithi089.ufybqw (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:05.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:04.354024+0000 mgr.smithi089.ufybqw (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T11:13:05.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:05.226895+0000 mon.smithi089 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.073+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f59f80fc890 con 0x7f59f80fab80 2024-01-21T11:13:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.075+0000 7f59e5ffb700 1 -- 172.21.15.89:0/2444676191 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f59e801b570 con 0x7f59f80fab80 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f59e002fe60 msgr2=0x7f59e0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f59e002fe60 0x7f59e0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f59f40096a0 tx=0x7f59f4006b40).stop 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 msgr2=0x7f59f810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f810a9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f59e800b1d0 tx=0x7f59e8006ee0).stop 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 shutdown_connections 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f59e002fe60 0x7f59e0032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 --2- 172.21.15.89:0/2444676191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f80fab80 0x7f59f810a9f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.078+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 >> 172.21.15.89:0/2444676191 conn(0x7f59f80f63c0 msgr2=0x7f59f8070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.079+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 shutdown_connections 2024-01-21T11:13:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.079+0000 7f59ff8f1700 1 -- 172.21.15.89:0/2444676191 wait complete. 2024-01-21T11:13:06.491 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T11:13:06.683 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:05.224097+0000 mgr.smithi089.ufybqw (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:06.683 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:05.224669+0000 mgr.smithi089.ufybqw (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T11:13:06.683 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:06.076362+0000 mon.smithi089 (mon.0) 81 : audit [INF] from='client.? 172.21.15.89:0/2444676191' entity='client.admin' 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.865+0000 7f8d6c671700 1 Processor -- start 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.865+0000 7f8d6c671700 1 -- start start 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6c671700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d64070a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6c671700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64072910 con 0x7f8d640724f0 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6a40d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d64070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6a40d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d64070a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60122/0 (socket says 172.21.15.89:60122) 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6a40d700 1 -- 172.21.15.89:0/4051067057 learned_addr learned my addr 172.21.15.89:0/4051067057 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6a40d700 1 -- 172.21.15.89:0/4051067057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d64070f80 con 0x7f8d640724f0 2024-01-21T11:13:06.869 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6a40d700 1 --2- 172.21.15.89:0/4051067057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d64070a90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8d5400b5f0 tx=0x7f8d54006ee0).ready entity=mon.0 client_cookie=17a31e0bcd872d5e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.866+0000 7f8d6940b700 1 -- 172.21.15.89:0/4051067057 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d5400d600 con 0x7f8d640724f0 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6940b700 1 -- 172.21.15.89:0/4051067057 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d5400dbd0 con 0x7f8d640724f0 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6940b700 1 -- 172.21.15.89:0/4051067057 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d54011c00 con 0x7f8d640724f0 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 -- 172.21.15.89:0/4051067057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 msgr2=0x7f8d64070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 --2- 172.21.15.89:0/4051067057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d64070a90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8d5400b5f0 tx=0x7f8d54006ee0).stop 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 -- 172.21.15.89:0/4051067057 shutdown_connections 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 --2- 172.21.15.89:0/4051067057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d64070a90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 -- 172.21.15.89:0/4051067057 >> 172.21.15.89:0/4051067057 conn(0x7f8d640f74c0 msgr2=0x7f8d640f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 -- 172.21.15.89:0/4051067057 shutdown_connections 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.867+0000 7f8d6c671700 1 -- 172.21.15.89:0/4051067057 wait complete. 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6c671700 1 Processor -- start 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6c671700 1 -- start start 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6c671700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d6410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6c671700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6410d230 con 0x7f8d640724f0 2024-01-21T11:13:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6a40d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d6410cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6a40d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d6410cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60126/0 (socket says 172.21.15.89:60126) 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6a40d700 1 -- 172.21.15.89:0/10868120 learned_addr learned my addr 172.21.15.89:0/10868120 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6a40d700 1 -- 172.21.15.89:0/10868120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5400b200 con 0x7f8d640724f0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.868+0000 7f8d6a40d700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d6410cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8d54006c70 tx=0x7f8d54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.869+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d5400da30 con 0x7f8d640724f0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.869+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d5401a9f0 con 0x7f8d640724f0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.869+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6410d430 con 0x7f8d640724f0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.869+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d5401b890 con 0x7f8d640724f0 2024-01-21T11:13:06.871 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.869+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6410d840 con 0x7f8d640724f0 2024-01-21T11:13:06.872 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.869+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8d5400f070 con 0x7f8d640724f0 2024-01-21T11:13:06.872 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.870+0000 7f8d52ffd700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8d4c02ff00 0x7f8d4c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:06.872 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.870+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8d540412d0 con 0x7f8d640724f0 2024-01-21T11:13:06.872 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.870+0000 7f8d69c0c700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8d4c02ff00 0x7f8d4c0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:06.872 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.870+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d64045ad0 con 0x7f8d640724f0 2024-01-21T11:13:06.872 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.870+0000 7f8d69c0c700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8d4c02ff00 0x7f8d4c0323c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8d600096a0 tx=0x7f8d60006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:06.874 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:06.874+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d54016070 con 0x7f8d640724f0 2024-01-21T11:13:07.029 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.029+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f8d64101de0 con 0x7f8d640724f0 2024-01-21T11:13:07.031 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.032+0000 7f8d52ffd700 1 -- 172.21.15.89:0/10868120 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8d64101de0 con 0x7f8d640724f0 2024-01-21T11:13:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.033+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8d4c02ff00 msgr2=0x7f8d4c0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.033+0000 7f8d6c671700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8d4c02ff00 0x7f8d4c0323c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8d600096a0 tx=0x7f8d60006b40).stop 2024-01-21T11:13:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 msgr2=0x7f8d6410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d6410cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8d54006c70 tx=0x7f8d54006ee0).stop 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 shutdown_connections 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8d4c02ff00 0x7f8d4c0323c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 --2- 172.21.15.89:0/10868120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d640724f0 0x7f8d6410cd40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 >> 172.21.15.89:0/10868120 conn(0x7f8d640f74c0 msgr2=0x7f8d640f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 shutdown_connections 2024-01-21T11:13:07.035 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.034+0000 7f8d6c671700 1 -- 172.21.15.89:0/10868120 wait complete. 2024-01-21T11:13:07.426 INFO:teuthology.orchestra.run.smithi089.stderr:Enabling the dashboard module... 2024-01-21T11:13:07.438 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T11:13:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe8250de700 1 Processor -- start 2024-01-21T11:13:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe8250de700 1 -- start start 2024-01-21T11:13:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe8250de700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe820100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe8250de700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8200fc750 con 0x7fe8200703b0 2024-01-21T11:13:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe81ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe820100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe81ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe820100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60142/0 (socket says 172.21.15.89:60142) 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.815+0000 7fe81ed9d700 1 -- 172.21.15.89:0/4236169826 learned_addr learned my addr 172.21.15.89:0/4236169826 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.816+0000 7fe81ed9d700 1 -- 172.21.15.89:0/4236169826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe820101250 con 0x7fe8200703b0 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.816+0000 7fe81ed9d700 1 --2- 172.21.15.89:0/4236169826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe820100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe81000b5f0 tx=0x7fe810006ee0).ready entity=mon.0 client_cookie=168ea8fc05b625a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.816+0000 7fe81dd9b700 1 -- 172.21.15.89:0/4236169826 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe81000d600 con 0x7fe8200703b0 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.816+0000 7fe81dd9b700 1 -- 172.21.15.89:0/4236169826 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe81000dc10 con 0x7fe8200703b0 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.816+0000 7fe81dd9b700 1 -- 172.21.15.89:0/4236169826 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe810011cd0 con 0x7fe8200703b0 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 -- 172.21.15.89:0/4236169826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 msgr2=0x7fe820100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 --2- 172.21.15.89:0/4236169826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe820100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe81000b5f0 tx=0x7fe810006ee0).stop 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 -- 172.21.15.89:0/4236169826 shutdown_connections 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 --2- 172.21.15.89:0/4236169826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe820100d60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 -- 172.21.15.89:0/4236169826 >> 172.21.15.89:0/4236169826 conn(0x7fe8200f63c0 msgr2=0x7fe8200f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 -- 172.21.15.89:0/4236169826 shutdown_connections 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.817+0000 7fe8250de700 1 -- 172.21.15.89:0/4236169826 wait complete. 2024-01-21T11:13:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe8250de700 1 Processor -- start 2024-01-21T11:13:07.820 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe8250de700 1 -- start start 2024-01-21T11:13:07.821 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe8250de700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe82010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe8250de700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82010d1d0 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe81ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe82010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe81ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe82010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60152/0 (socket says 172.21.15.89:60152) 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.818+0000 7fe81ed9d700 1 -- 172.21.15.89:0/2631068106 learned_addr learned my addr 172.21.15.89:0/2631068106 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe81ed9d700 1 -- 172.21.15.89:0/2631068106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81000b200 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe81ed9d700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe82010cce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe81000b1d0 tx=0x7fe810006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe810011df0 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe81001b8c0 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82010d3d0 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe81001a830 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.819+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82010d7e0 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.820+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe81000f070 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.820+0000 7fe8077fe700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fe80c02feb0 0x7fe80c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.820+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe810040f80 con 0x7fe8200703b0 2024-01-21T11:13:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.820+0000 7fe81e59c700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fe80c02feb0 0x7fe80c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:07.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.820+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe820101e20 con 0x7fe8200703b0 2024-01-21T11:13:07.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.821+0000 7fe81e59c700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fe80c02feb0 0x7fe80c032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe8140096a0 tx=0x7fe814006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:07.824 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:07.825+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe810016070 con 0x7fe8200703b0 2024-01-21T11:13:08.029 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.029+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe82010e610 con 0x7fe8200703b0 2024-01-21T11:13:08.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:07.032534+0000 mon.smithi089 (mon.0) 82 : audit [INF] from='client.? 172.21.15.89:0/10868120' entity='client.admin' 2024-01-21T11:13:08.434 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:07.399696+0000 mon.smithi089 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:08.434 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:07.626160+0000 mon.smithi089 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:08.434 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:08.030153+0000 mon.smithi089 (mon.0) 85 : audit [INF] from='client.? 172.21.15.89:0/2631068106' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T11:13:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.629+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fe810024790 con 0x7fe8200703b0 2024-01-21T11:13:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.630+0000 7fe8077fe700 1 -- 172.21.15.89:0/2631068106 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe810040640 con 0x7fe8200703b0 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fe80c02feb0 msgr2=0x7fe80c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fe80c02feb0 0x7fe80c032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe8140096a0 tx=0x7fe814006b40).stop 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 msgr2=0x7fe82010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe82010cce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe81000b1d0 tx=0x7fe810006ee0).stop 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 shutdown_connections 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7fe80c02feb0 0x7fe80c032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 --2- 172.21.15.89:0/2631068106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8200703b0 0x7fe82010cce0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 >> 172.21.15.89:0/2631068106 conn(0x7fe8200f63c0 msgr2=0x7fe8200f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:08.633 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 shutdown_connections 2024-01-21T11:13:08.634 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:08.632+0000 7fe8250de700 1 -- 172.21.15.89:0/2631068106 wait complete. 2024-01-21T11:13:09.259 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T11:13:09.408 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:09 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:08.265871+0000 mon.smithi089 (mon.0) 86 : audit [INF] from='mgr.14142 172.21.15.89:0/2375686149' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:09.409 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:09 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:08.630711+0000 mon.smithi089 (mon.0) 87 : audit [INF] from='client.? 172.21.15.89:0/2631068106' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T11:13:09.409 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:09 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:08.630762+0000 mon.smithi089 (mon.0) 88 : cluster [DBG] mgrmap e12: smithi089.ufybqw(active, since 7s) 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bbda0a700 1 Processor -- start 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bbda0a700 1 -- start start 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bbda0a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bbda0a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80fc750 con 0x7f8bb80703b0 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bb6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bb6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60178/0 (socket says 172.21.15.89:60178) 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bb6ffd700 1 -- 172.21.15.89:0/343756560 learned_addr learned my addr 172.21.15.89:0/343756560 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.618+0000 7f8bb6ffd700 1 -- 172.21.15.89:0/343756560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8101250 con 0x7f8bb80703b0 2024-01-21T11:13:09.619 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.619+0000 7f8bb6ffd700 1 --2- 172.21.15.89:0/343756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb8100d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8ba800b5f0 tx=0x7f8ba8006ee0).ready entity=mon.0 client_cookie=77c8d3eb648087b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:09.620 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.619+0000 7f8bb5ffb700 1 -- 172.21.15.89:0/343756560 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba800d600 con 0x7f8bb80703b0 2024-01-21T11:13:09.620 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.619+0000 7f8bb5ffb700 1 -- 172.21.15.89:0/343756560 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ba800dc10 con 0x7f8bb80703b0 2024-01-21T11:13:09.620 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.619+0000 7f8bb5ffb700 1 -- 172.21.15.89:0/343756560 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba8011cd0 con 0x7f8bb80703b0 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.619+0000 7f8bbda0a700 1 -- 172.21.15.89:0/343756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 msgr2=0x7f8bb8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.619+0000 7f8bbda0a700 1 --2- 172.21.15.89:0/343756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb8100d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8ba800b5f0 tx=0x7f8ba8006ee0).stop 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 -- 172.21.15.89:0/343756560 shutdown_connections 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 --2- 172.21.15.89:0/343756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb8100d60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 -- 172.21.15.89:0/343756560 >> 172.21.15.89:0/343756560 conn(0x7f8bb80f63c0 msgr2=0x7f8bb80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 -- 172.21.15.89:0/343756560 shutdown_connections 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 -- 172.21.15.89:0/343756560 wait complete. 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 Processor -- start 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 -- start start 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.620+0000 7f8bbda0a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb810d1d0 con 0x7f8bb80703b0 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bb6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:09.622 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bb6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60180/0 (socket says 172.21.15.89:60180) 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bb6ffd700 1 -- 172.21.15.89:0/3119407016 learned_addr learned my addr 172.21.15.89:0/3119407016 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bb6ffd700 1 -- 172.21.15.89:0/3119407016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba800b200 con 0x7f8bb80703b0 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bb6ffd700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb810cce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8ba800b1d0 tx=0x7f8ba8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba8011df0 con 0x7f8bb80703b0 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb810d3d0 con 0x7f8bb80703b0 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ba801b8c0 con 0x7f8bb80703b0 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba801a830 con 0x7f8bb80703b0 2024-01-21T11:13:09.623 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.621+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb810d7e0 con 0x7f8bb80703b0 2024-01-21T11:13:09.624 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.622+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb8045ad0 con 0x7f8bb80703b0 2024-01-21T11:13:09.624 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.623+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8ba800f070 con 0x7f8bb80703b0 2024-01-21T11:13:09.624 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.623+0000 7f8b9f7fe700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8ba402feb0 0x7f8ba4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:09.624 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.623+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8ba8041ac0 con 0x7f8bb80703b0 2024-01-21T11:13:09.624 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.623+0000 7f8bb67fc700 1 -- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8ba402feb0 msgr2=0x7f8ba4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/578651808 2024-01-21T11:13:09.624 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.623+0000 7f8bb67fc700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8ba402feb0 0x7f8ba4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:13:09.627 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.627+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ba80239c0 con 0x7f8bb80703b0 2024-01-21T11:13:09.820 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.819+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8bb8106a40 con 0x7f8bb80703b0 2024-01-21T11:13:09.820 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.819+0000 7f8b9f7fe700 1 -- 172.21.15.89:0/3119407016 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f8ba8016070 con 0x7f8bb80703b0 2024-01-21T11:13:09.820 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:13:09.820 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "epoch": 12, 2024-01-21T11:13:09.820 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "available": true, 2024-01-21T11:13:09.821 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "active_name": "smithi089.ufybqw", 2024-01-21T11:13:09.821 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T11:13:09.821 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8ba402feb0 msgr2=0x7f8ba4032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8ba402feb0 0x7f8ba4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 msgr2=0x7f8bb810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb810cce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8ba800b1d0 tx=0x7f8ba8006ee0).stop 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 shutdown_connections 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f8ba402feb0 0x7f8ba4032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 --2- 172.21.15.89:0/3119407016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bb80703b0 0x7f8bb810cce0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 >> 172.21.15.89:0/3119407016 conn(0x7f8bb80f63c0 msgr2=0x7f8bb80f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:09.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 shutdown_connections 2024-01-21T11:13:09.823 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:09.821+0000 7f8bbda0a700 1 -- 172.21.15.89:0/3119407016 wait complete. 2024-01-21T11:13:10.234 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for the mgr to restart... 2024-01-21T11:13:10.234 INFO:teuthology.orchestra.run.smithi089.stderr:Waiting for mgr epoch 12... 2024-01-21T11:13:10.234 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f30a2b700 1 Processor -- start 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f30a2b700 1 -- start start 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f30a2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f30a2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c0fec20 con 0x7f5f2c070430 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f2ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f2ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60194/0 (socket says 172.21.15.89:60194) 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f2ad9d700 1 -- 172.21.15.89:0/134102692 learned_addr learned my addr 172.21.15.89:0/134102692 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:10.607 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f2ad9d700 1 -- 172.21.15.89:0/134102692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c0fed60 con 0x7f5f2c070430 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.606+0000 7f5f2ad9d700 1 --2- 172.21.15.89:0/134102692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c0fe730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5f1c00b5f0 tx=0x7f5f1c006ee0).ready entity=mon.0 client_cookie=63913520162a69a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f29d9b700 1 -- 172.21.15.89:0/134102692 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c00d600 con 0x7f5f2c070430 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f29d9b700 1 -- 172.21.15.89:0/134102692 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f1c00dc10 con 0x7f5f2c070430 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f29d9b700 1 -- 172.21.15.89:0/134102692 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c011cd0 con 0x7f5f2c070430 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 -- 172.21.15.89:0/134102692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 msgr2=0x7f5f2c0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/134102692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c0fe730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5f1c00b5f0 tx=0x7f5f1c006ee0).stop 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 -- 172.21.15.89:0/134102692 shutdown_connections 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/134102692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c0fe730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 -- 172.21.15.89:0/134102692 >> 172.21.15.89:0/134102692 conn(0x7f5f2c0f6210 msgr2=0x7f5f2c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:10.611 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 -- 172.21.15.89:0/134102692 shutdown_connections 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.607+0000 7f5f30a2b700 1 -- 172.21.15.89:0/134102692 wait complete. 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f30a2b700 1 Processor -- start 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f30a2b700 1 -- start start 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f30a2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c10a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f30a2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c10ae00 con 0x7f5f2c070430 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f2ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c10a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f2ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c10a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60206/0 (socket says 172.21.15.89:60206) 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.608+0000 7f5f2ad9d700 1 -- 172.21.15.89:0/2043265988 learned_addr learned my addr 172.21.15.89:0/2043265988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f2ad9d700 1 -- 172.21.15.89:0/2043265988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c00b200 con 0x7f5f2c070430 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f2ad9d700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c10a910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5f1c00df90 tx=0x7f5f1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c011da0 con 0x7f5f2c070430 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f1c01b8c0 con 0x7f5f2c070430 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c10b000 con 0x7f5f2c070430 2024-01-21T11:13:10.612 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c01a930 con 0x7f5f2c070430 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.609+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c10b410 con 0x7f5f2c070430 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.610+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f5f1c00f070 con 0x7f5f2c070430 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.610+0000 7f5f137fe700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.610+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 --> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f18032a80 con 0x7f5f1802ff00 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.610+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5f1c041080 con 0x7f5f2c070430 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.610+0000 7f5f2a59c700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 msgr2=0x7f5f180323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/578651808 2024-01-21T11:13:10.613 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.610+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:13:10.811 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.811+0000 7f5f2a59c700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 msgr2=0x7f5f180323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/578651808 2024-01-21T11:13:10.811 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:10.811+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:13:10.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:09.820282+0000 mon.smithi089 (mon.0) 89 : audit [DBG] from='client.? 172.21.15.89:0/3119407016' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T11:13:11.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:11.212+0000 7f5f2a59c700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 msgr2=0x7f5f180323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/578651808 2024-01-21T11:13:11.212 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:11.212+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T11:13:12.013 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:12.013+0000 7f5f2a59c700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 msgr2=0x7f5f180323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/578651808 2024-01-21T11:13:12.014 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:12.013+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T11:13:13.615 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:13.614+0000 7f5f2a59c700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 msgr2=0x7f5f180323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/578651808 2024-01-21T11:13:13.615 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:13.614+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T11:13:13.635 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:13.635+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f5f1c010dd0 con 0x7f5f2c070430 2024-01-21T11:13:13.635 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:13.635+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 msgr2=0x7f5f180323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:13:13.635 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:13.635+0000 7f5f137fe700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:13:13.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:13.584350+0000 mon.smithi089 (mon.0) 90 : cluster [INF] Active manager daemon smithi089.ufybqw restarted 2024-01-21T11:13:13.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:13.584744+0000 mon.smithi089 (mon.0) 91 : cluster [INF] Activating manager daemon smithi089.ufybqw 2024-01-21T11:13:13.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:13.586359+0000 mon.smithi089 (mon.0) 92 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T11:13:14.650 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.649+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f5f1c019ce0 con 0x7f5f2c070430 2024-01-21T11:13:14.651 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.649+0000 7f5f137fe700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5f18033b90 0x7f5f18035f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:14.651 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.649+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f18032a80 con 0x7f5f18033b90 2024-01-21T11:13:14.651 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.650+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5f18033b90 0x7f5f18035f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:14.651 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.650+0000 7f5f2a59c700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5f18033b90 0x7f5f18035f80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5f20006fd0 tx=0x7f5f20006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:14.651 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.650+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mgr.14162 v2:172.21.15.89:6800/111466865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f5f18032a80 con 0x7f5f18033b90 2024-01-21T11:13:14.656 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.656+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5f2c059100 con 0x7f5f18033b90 2024-01-21T11:13:14.656 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f137fe700 1 -- 172.21.15.89:0/2043265988 <== mgr.14162 v2:172.21.15.89:6800/111466865 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5f2c059100 con 0x7f5f18033b90 2024-01-21T11:13:14.656 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: { 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: } 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5f18033b90 msgr2=0x7f5f18035f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5f18033b90 0x7f5f18035f80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5f20006fd0 tx=0x7f5f20006d70).stop 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 msgr2=0x7f5f2c10a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c10a910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5f1c00df90 tx=0x7f5f1c006ee0).stop 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 shutdown_connections 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5f18033b90 0x7f5f18035f80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c070430 0x7f5f2c10a910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 --2- 172.21.15.89:0/2043265988 >> [v2:172.21.15.89:6800/578651808,v1:172.21.15.89:6801/578651808] conn(0x7f5f1802ff00 0x7f5f180323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 >> 172.21.15.89:0/2043265988 conn(0x7f5f2c0f6210 msgr2=0x7f5f2c0f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:14.658 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 shutdown_connections 2024-01-21T11:13:14.659 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:14.657+0000 7f5f30a2b700 1 -- 172.21.15.89:0/2043265988 wait complete. 2024-01-21T11:13:14.833 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:13.637224+0000 mon.smithi089 (mon.0) 93 : cluster [DBG] mgrmap e13: smithi089.ufybqw(active, starting, since 0.0525647s) 2024-01-21T11:13:14.833 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.638107+0000 mon.smithi089 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-01-21T11:13:14.833 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.638611+0000 mon.smithi089 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mgr metadata", "who": "smithi089.ufybqw", "id": "smithi089.ufybqw"}]: dispatch 2024-01-21T11:13:14.833 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.639217+0000 mon.smithi089 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:13:14.833 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.639387+0000 mon.smithi089 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:13:14.833 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.639570+0000 mon.smithi089 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:13:14.834 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:13.653100+0000 mon.smithi089 (mon.0) 99 : cluster [INF] Manager daemon smithi089.ufybqw is now available 2024-01-21T11:13:14.834 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.751560+0000 mon.smithi089 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:13:14.834 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.753786+0000 mon.smithi089 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:13:14.834 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.769643+0000 mon.smithi089 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:13:14.834 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:13.778690+0000 mon.smithi089 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:13:14.834 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:14.580167+0000 mon.smithi089 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:15.017 INFO:teuthology.orchestra.run.smithi089.stderr:mgr epoch 12 is available 2024-01-21T11:13:15.017 INFO:teuthology.orchestra.run.smithi089.stderr:Generating a dashboard self-signed certificate... 2024-01-21T11:13:15.017 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T11:13:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.408+0000 7f03199ec700 1 Processor -- start 2024-01-21T11:13:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.408+0000 7f03199ec700 1 -- start start 2024-01-21T11:13:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.408+0000 7f03199ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f0314100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.408+0000 7f03199ec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03140fc580 con 0x7f03140703d0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03189ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f0314100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03189ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f0314100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41006/0 (socket says 172.21.15.89:41006) 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03189ea700 1 -- 172.21.15.89:0/2635847902 learned_addr learned my addr 172.21.15.89:0/2635847902 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03189ea700 1 -- 172.21.15.89:0/2635847902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314101080 con 0x7f03140703d0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03189ea700 1 --2- 172.21.15.89:0/2635847902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f0314100b90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f030800b5f0 tx=0x7f0308006ee0).ready entity=mon.0 client_cookie=187e7217c8f3768 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03137fe700 1 -- 172.21.15.89:0/2635847902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f030800d600 con 0x7f03140703d0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03137fe700 1 -- 172.21.15.89:0/2635847902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f030800dc10 con 0x7f03140703d0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.409+0000 7f03137fe700 1 -- 172.21.15.89:0/2635847902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0308011cd0 con 0x7f03140703d0 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 -- 172.21.15.89:0/2635847902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 msgr2=0x7f0314100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 --2- 172.21.15.89:0/2635847902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f0314100b90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f030800b5f0 tx=0x7f0308006ee0).stop 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 -- 172.21.15.89:0/2635847902 shutdown_connections 2024-01-21T11:13:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 --2- 172.21.15.89:0/2635847902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f0314100b90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 -- 172.21.15.89:0/2635847902 >> 172.21.15.89:0/2635847902 conn(0x7f03140f6210 msgr2=0x7f03140f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 -- 172.21.15.89:0/2635847902 shutdown_connections 2024-01-21T11:13:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.410+0000 7f03199ec700 1 -- 172.21.15.89:0/2635847902 wait complete. 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03199ec700 1 Processor -- start 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03199ec700 1 -- start start 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03199ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f031410ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03199ec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031410cf70 con 0x7f03140703d0 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03189ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f031410ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03189ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f031410ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41022/0 (socket says 172.21.15.89:41022) 2024-01-21T11:13:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.411+0000 7f03189ea700 1 -- 172.21.15.89:0/2058524383 learned_addr learned my addr 172.21.15.89:0/2058524383 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03189ea700 1 -- 172.21.15.89:0/2058524383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030800b200 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03189ea700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f031410ca80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0308006a40 tx=0x7f0308006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0308011c90 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f030801b8c0 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031410d170 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f030801a8c0 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.412+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031410d580 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.413+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f030800f070 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.413+0000 7f03117fa700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802fdc0 0x7f02f8032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.413+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0308040920 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.413+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0314045ad0 con 0x7f03140703d0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.413+0000 7f0313fff700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802fdc0 0x7f02f8032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.413+0000 7f0313fff700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802fdc0 0x7f02f8032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f03000096a0 tx=0x7f0300006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:15.417 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.418+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0308016070 con 0x7f03140703d0 2024-01-21T11:13:15.574 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.574+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f0314106a20 con 0x7f02f802fdc0 2024-01-21T11:13:15.650 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.650+0000 7f03117fa700 1 -- 172.21.15.89:0/2058524383 <== mgr.14162 v2:172.21.15.89:6800/111466865 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f0314106a20 con 0x7f02f802fdc0 2024-01-21T11:13:15.650 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T11:13:15.652 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802fdc0 msgr2=0x7f02f8032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:15.652 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802fdc0 0x7f02f8032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f03000096a0 tx=0x7f0300006b40).stop 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 msgr2=0x7f031410ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f031410ca80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0308006a40 tx=0x7f0308006ee0).stop 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 shutdown_connections 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802fdc0 0x7f02f8032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 --2- 172.21.15.89:0/2058524383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03140703d0 0x7f031410ca80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 >> 172.21.15.89:0/2058524383 conn(0x7f03140f6210 msgr2=0x7f03140f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 shutdown_connections 2024-01-21T11:13:15.653 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:15.652+0000 7f03199ec700 1 -- 172.21.15.89:0/2058524383 wait complete. 2024-01-21T11:13:15.849 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:14.651186+0000 mgr.smithi089.ufybqw (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T11:13:15.849 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:14.651717+0000 mon.smithi089 (mon.0) 105 : cluster [DBG] mgrmap e14: smithi089.ufybqw(active, since 1.06706s) 2024-01-21T11:13:15.849 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:14.657356+0000 mgr.smithi089.ufybqw (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T11:13:15.849 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:15.646303+0000 mon.smithi089 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:16.066 INFO:teuthology.orchestra.run.smithi089.stderr:Creating initial admin user... 2024-01-21T11:13:16.066 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0g_dowqd:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T11:13:16.382 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.379+0000 7f43df540700 1 Processor -- start 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.379+0000 7f43df540700 1 -- start start 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43df540700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d8100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43df540700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80fc730 con 0x7f43d80703d0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43dd2dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d8100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43dd2dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d8100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41034/0 (socket says 172.21.15.89:41034) 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43dd2dc700 1 -- 172.21.15.89:0/3073959433 learned_addr learned my addr 172.21.15.89:0/3073959433 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43dd2dc700 1 -- 172.21.15.89:0/3073959433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d8101230 con 0x7f43d80703d0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43dd2dc700 1 --2- 172.21.15.89:0/3073959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d8100d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f43c800b5f0 tx=0x7f43c8006ee0).ready entity=mon.0 client_cookie=d97f7e2a9c6de905 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43c7fff700 1 -- 172.21.15.89:0/3073959433 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43c800d600 con 0x7f43d80703d0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43c7fff700 1 -- 172.21.15.89:0/3073959433 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43c800dc10 con 0x7f43d80703d0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.380+0000 7f43c7fff700 1 -- 172.21.15.89:0/3073959433 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43c8011cd0 con 0x7f43d80703d0 2024-01-21T11:13:16.383 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 -- 172.21.15.89:0/3073959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 msgr2=0x7f43d8100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 --2- 172.21.15.89:0/3073959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d8100d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f43c800b5f0 tx=0x7f43c8006ee0).stop 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 -- 172.21.15.89:0/3073959433 shutdown_connections 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 --2- 172.21.15.89:0/3073959433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d8100d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 -- 172.21.15.89:0/3073959433 >> 172.21.15.89:0/3073959433 conn(0x7f43d80f63c0 msgr2=0x7f43d80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 -- 172.21.15.89:0/3073959433 shutdown_connections 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 -- 172.21.15.89:0/3073959433 wait complete. 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.381+0000 7f43df540700 1 Processor -- start 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43df540700 1 -- start start 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43df540700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43df540700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d810d1c0 con 0x7f43d80703d0 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43dd2dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43dd2dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41050/0 (socket says 172.21.15.89:41050) 2024-01-21T11:13:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43dd2dc700 1 -- 172.21.15.89:0/971407404 learned_addr learned my addr 172.21.15.89:0/971407404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43dd2dc700 1 -- 172.21.15.89:0/971407404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c800b200 con 0x7f43d80703d0 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43dd2dc700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d810ccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f43c8006a40 tx=0x7f43c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43c8011c90 con 0x7f43d80703d0 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43c801b8c0 con 0x7f43d80703d0 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43c801a8c0 con 0x7f43d80703d0 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.382+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d810d3c0 con 0x7f43d80703d0 2024-01-21T11:13:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.383+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d810d7d0 con 0x7f43d80703d0 2024-01-21T11:13:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.383+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f43c800f070 con 0x7f43d80703d0 2024-01-21T11:13:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.383+0000 7f43c5ffb700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f43c002ff00 0x7f43c00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.384+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43c8040a10 con 0x7f43d80703d0 2024-01-21T11:13:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.384+0000 7f43dcadb700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f43c002ff00 0x7f43c00323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.384+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d8101e00 con 0x7f43d80703d0 2024-01-21T11:13:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.384+0000 7f43dcadb700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f43c002ff00 0x7f43c00323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f43d40096a0 tx=0x7f43d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:16.387 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.388+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43c8016070 con 0x7f43d80703d0 2024-01-21T11:13:16.553 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.553+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f43d8057f10 con 0x7f43c002ff00 2024-01-21T11:13:16.769 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.769+0000 7f43c5ffb700 1 -- 172.21.15.89:0/971407404 <== mgr.14162 v2:172.21.15.89:6800/111466865 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f43d8057f10 con 0x7f43c002ff00 2024-01-21T11:13:16.769 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$v.81M62xnvOlu8N0v3s.TO0vjkRT8hhxlM5j6UZeeWXIxQK737E8O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705835596, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T11:13:16.771 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f43c002ff00 msgr2=0x7f43c00323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f43c002ff00 0x7f43c00323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f43d40096a0 tx=0x7f43d4006b40).stop 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 msgr2=0x7f43d810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d810ccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f43c8006a40 tx=0x7f43c8006ee0).stop 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 shutdown_connections 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f43c002ff00 0x7f43c00323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 --2- 172.21.15.89:0/971407404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80703d0 0x7f43d810ccd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 >> 172.21.15.89:0/971407404 conn(0x7f43d80f63c0 msgr2=0x7f43d80f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 shutdown_connections 2024-01-21T11:13:16.772 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:16.771+0000 7f43df540700 1 -- 172.21.15.89:0/971407404 wait complete. 2024-01-21T11:13:16.924 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:15.575933+0000 mgr.smithi089.ufybqw (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:16.924 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:15.650667+0000 mon.smithi089 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:16.924 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:15.656048+0000 mon.smithi089 (mon.0) 108 : cluster [DBG] mgrmap e15: smithi089.ufybqw(active, since 2s) 2024-01-21T11:13:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:Fetching dashboard port number... 2024-01-21T11:13:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp_kl5xgo3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpr76ygqdl:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T11:13:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.479+0000 7f833cce2700 1 Processor -- start 2024-01-21T11:13:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.479+0000 7f833cce2700 1 -- start start 2024-01-21T11:13:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833cce2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833cce2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380fb490 con 0x7f83380fab80 2024-01-21T11:13:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41052/0 (socket says 172.21.15.89:41052) 2024-01-21T11:13:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833659c700 1 -- 172.21.15.89:0/3569941390 learned_addr learned my addr 172.21.15.89:0/3569941390 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833659c700 1 -- 172.21.15.89:0/3569941390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380fb5d0 con 0x7f83380fab80 2024-01-21T11:13:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833659c700 1 --2- 172.21.15.89:0/3569941390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f832c00b5f0 tx=0x7f832c006ee0).ready entity=mon.0 client_cookie=2f33a7099368d8f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.480+0000 7f833559a700 1 -- 172.21.15.89:0/3569941390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c00d600 con 0x7f83380fab80 2024-01-21T11:13:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833559a700 1 -- 172.21.15.89:0/3569941390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f832c00dc10 con 0x7f83380fab80 2024-01-21T11:13:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833559a700 1 -- 172.21.15.89:0/3569941390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c011cd0 con 0x7f83380fab80 2024-01-21T11:13:17.490 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 -- 172.21.15.89:0/3569941390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 msgr2=0x7f83380fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:17.490 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 --2- 172.21.15.89:0/3569941390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f832c00b5f0 tx=0x7f832c006ee0).stop 2024-01-21T11:13:17.490 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 -- 172.21.15.89:0/3569941390 shutdown_connections 2024-01-21T11:13:17.490 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 --2- 172.21.15.89:0/3569941390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83380fafa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 -- 172.21.15.89:0/3569941390 >> 172.21.15.89:0/3569941390 conn(0x7f83380f63c0 msgr2=0x7f83380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 -- 172.21.15.89:0/3569941390 shutdown_connections 2024-01-21T11:13:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.481+0000 7f833cce2700 1 -- 172.21.15.89:0/3569941390 wait complete. 2024-01-21T11:13:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833cce2700 1 Processor -- start 2024-01-21T11:13:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833cce2700 1 -- start start 2024-01-21T11:13:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833cce2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83381088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833cce2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338108d90 con 0x7f83380fab80 2024-01-21T11:13:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83381088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83381088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41064/0 (socket says 172.21.15.89:41064) 2024-01-21T11:13:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833659c700 1 -- 172.21.15.89:0/2402281770 learned_addr learned my addr 172.21.15.89:0/2402281770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.482+0000 7f833659c700 1 -- 172.21.15.89:0/2402281770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c00b200 con 0x7f83380fab80 2024-01-21T11:13:17.493 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.483+0000 7f833659c700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83381088a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f832c00dfb0 tx=0x7f832c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:17.493 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.483+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c011df0 con 0x7f83380fab80 2024-01-21T11:13:17.493 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.483+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f832c01b8c0 con 0x7f83380fab80 2024-01-21T11:13:17.493 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.483+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8338108f90 con 0x7f83380fab80 2024-01-21T11:13:17.493 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.483+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c01a8c0 con 0x7f83380fab80 2024-01-21T11:13:17.493 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.483+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83381093a0 con 0x7f83380fab80 2024-01-21T11:13:17.494 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.484+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f832c00f070 con 0x7f83380fab80 2024-01-21T11:13:17.494 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.484+0000 7f8326ffd700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f831c032100 0x7f831c0345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:17.494 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.484+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f832c041170 con 0x7f83380fab80 2024-01-21T11:13:17.494 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.484+0000 7f8335d9b700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f831c032100 0x7f831c0345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:17.494 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.484+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8338059100 con 0x7f83380fab80 2024-01-21T11:13:17.494 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.484+0000 7f8335d9b700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f831c032100 0x7f831c0345c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f83280096a0 tx=0x7f8328006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:17.495 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.488+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f832c016070 con 0x7f83380fab80 2024-01-21T11:13:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.637+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f8338045ad0 con 0x7f83380fab80 2024-01-21T11:13:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.637+0000 7f8326ffd700 1 -- 172.21.15.89:0/2402281770 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f832c010360 con 0x7f83380fab80 2024-01-21T11:13:17.638 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 8443 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f831c032100 msgr2=0x7f831c0345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f831c032100 0x7f831c0345c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f83280096a0 tx=0x7f8328006b40).stop 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 msgr2=0x7f83381088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83381088a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f832c00dfb0 tx=0x7f832c006ee0).stop 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 shutdown_connections 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f831c032100 0x7f831c0345c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 --2- 172.21.15.89:0/2402281770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83380fab80 0x7f83381088a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 >> 172.21.15.89:0/2402281770 conn(0x7f83380f63c0 msgr2=0x7f83380ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 shutdown_connections 2024-01-21T11:13:17.640 INFO:teuthology.orchestra.run.smithi089.stderr:/usr/bin/ceph: 2024-01-21T11:13:17.639+0000 7f833cce2700 1 -- 172.21.15.89:0/2402281770 wait complete. 2024-01-21T11:13:17.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:16.554638+0000 mgr.smithi089.ufybqw (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:17.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:16.769643+0000 mon.smithi089 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:17.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:17.638276+0000 mon.smithi089 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.89:0/2402281770' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T11:13:18.116 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:13:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: disabled 2024-01-21T11:13:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:13:18.131 INFO:teuthology.orchestra.run.smithi089.stderr:systemctl: inactive 2024-01-21T11:13:18.132 INFO:teuthology.orchestra.run.smithi089.stderr:firewalld.service is not enabled 2024-01-21T11:13:18.132 INFO:teuthology.orchestra.run.smithi089.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr:Ceph Dashboard is now available at: 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: URL: https://smithi089.front.sepia.ceph.com:8443/ 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: User: admin 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: Password: mc3c7lljmz 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr:You can access the Ceph CLI with: 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T11:13:18.135 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr: ceph telemetry on 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr:For more information see: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr:Bootstrap complete. 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr:Releasing lock 140106901137896 on /run/cephadm/e44d1d4c-b84d-11ee-95b0-87774f69a715.lock 2024-01-21T11:13:18.136 INFO:teuthology.orchestra.run.smithi089.stderr:Lock 140106901137896 released on /run/cephadm/e44d1d4c-b84d-11ee-95b0-87774f69a715.lock 2024-01-21T11:13:18.173 INFO:tasks.cephadm:Fetching config... 2024-01-21T11:13:18.173 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:13:18.173 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T11:13:18.233 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T11:13:18.233 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:13:18.233 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T11:13:18.294 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T11:13:18.294 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:13:18.295 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/mon.smithi089/keyring of=/dev/stdout 2024-01-21T11:13:18.362 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T11:13:18.362 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-01-21T11:13:18.362 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T11:13:18.420 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T11:13:18.421 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6RXUIL8Xg+b0/51yl7ZAP3B9qqPCEfNMYVGtf8CQT29EmHL8rYqt/z4gQo/jgGPVNj7X4BVfQu7zL6RcCaB1204imTZXTnwzUYNixZPD2iB4fHQ21RDyRfZACoHztyo/naOP8mm6MZHwrXgEjjAScnbDpWGou5h46B3N1YLyFlAqHLYx3cQ8rlu4xgpQGDl8+5EzICawC1KyMsuDLawaFAh5TFQnOCVAitSJgZ8c3ZGnNBtAEQ0HmDa5f1BWh8+ZOr7jJa/rxdhrQ7LBOFNUVvMZCFhmkQrO8fu6gjZUmzlbia2fYojH7FwqELPZNY6qVQCyT1Pc/ELn0CzdfIJKb/Ku9FfDraOdONjLrmwlgo5Y1+bGH13X5oBQwvurn5nevo/Z2uPJ56hn+ygPShA0cMTNURjStOWS/ra6yd6AMtKHZCd/BHtj8Pc6atfpiLYDMkiwdTAE+kAGI+ISmDsekTqjG1lCoMtmiMCNUHNhr2sicaCR16asDfwtdJXP/8h8= ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T11:13:18.497 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6RXUIL8Xg+b0/51yl7ZAP3B9qqPCEfNMYVGtf8CQT29EmHL8rYqt/z4gQo/jgGPVNj7X4BVfQu7zL6RcCaB1204imTZXTnwzUYNixZPD2iB4fHQ21RDyRfZACoHztyo/naOP8mm6MZHwrXgEjjAScnbDpWGou5h46B3N1YLyFlAqHLYx3cQ8rlu4xgpQGDl8+5EzICawC1KyMsuDLawaFAh5TFQnOCVAitSJgZ8c3ZGnNBtAEQ0HmDa5f1BWh8+ZOr7jJa/rxdhrQ7LBOFNUVvMZCFhmkQrO8fu6gjZUmzlbia2fYojH7FwqELPZNY6qVQCyT1Pc/ELn0CzdfIJKb/Ku9FfDraOdONjLrmwlgo5Y1+bGH13X5oBQwvurn5nevo/Z2uPJ56hn+ygPShA0cMTNURjStOWS/ra6yd6AMtKHZCd/BHtj8Pc6atfpiLYDMkiwdTAE+kAGI+ISmDsekTqjG1lCoMtmiMCNUHNhr2sicaCR16asDfwtdJXP/8h8= ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:13:18.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6RXUIL8Xg+b0/51yl7ZAP3B9qqPCEfNMYVGtf8CQT29EmHL8rYqt/z4gQo/jgGPVNj7X4BVfQu7zL6RcCaB1204imTZXTnwzUYNixZPD2iB4fHQ21RDyRfZACoHztyo/naOP8mm6MZHwrXgEjjAScnbDpWGou5h46B3N1YLyFlAqHLYx3cQ8rlu4xgpQGDl8+5EzICawC1KyMsuDLawaFAh5TFQnOCVAitSJgZ8c3ZGnNBtAEQ0HmDa5f1BWh8+ZOr7jJa/rxdhrQ7LBOFNUVvMZCFhmkQrO8fu6gjZUmzlbia2fYojH7FwqELPZNY6qVQCyT1Pc/ELn0CzdfIJKb/Ku9FfDraOdONjLrmwlgo5Y1+bGH13X5oBQwvurn5nevo/Z2uPJ56hn+ygPShA0cMTNURjStOWS/ra6yd6AMtKHZCd/BHtj8Pc6atfpiLYDMkiwdTAE+kAGI+ISmDsekTqjG1lCoMtmiMCNUHNhr2sicaCR16asDfwtdJXP/8h8= ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T11:13:18.553 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6RXUIL8Xg+b0/51yl7ZAP3B9qqPCEfNMYVGtf8CQT29EmHL8rYqt/z4gQo/jgGPVNj7X4BVfQu7zL6RcCaB1204imTZXTnwzUYNixZPD2iB4fHQ21RDyRfZACoHztyo/naOP8mm6MZHwrXgEjjAScnbDpWGou5h46B3N1YLyFlAqHLYx3cQ8rlu4xgpQGDl8+5EzICawC1KyMsuDLawaFAh5TFQnOCVAitSJgZ8c3ZGnNBtAEQ0HmDa5f1BWh8+ZOr7jJa/rxdhrQ7LBOFNUVvMZCFhmkQrO8fu6gjZUmzlbia2fYojH7FwqELPZNY6qVQCyT1Pc/ELn0CzdfIJKb/Ku9FfDraOdONjLrmwlgo5Y1+bGH13X5oBQwvurn5nevo/Z2uPJ56hn+ygPShA0cMTNURjStOWS/ra6yd6AMtKHZCd/BHtj8Pc6atfpiLYDMkiwdTAE+kAGI+ISmDsekTqjG1lCoMtmiMCNUHNhr2sicaCR16asDfwtdJXP/8h8= ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715 2024-01-21T11:13:18.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 -- 172.21.15.89:0/1307167661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 msgr2=0x7f509c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 --2- 172.21.15.89:0/1307167661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f90f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f5090009fb0 tx=0x7f5090009b30).stop 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 -- 172.21.15.89:0/1307167661 shutdown_connections 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 --2- 172.21.15.89:0/1307167661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f90f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 -- 172.21.15.89:0/1307167661 >> 172.21.15.89:0/1307167661 conn(0x7f509c0f0b70 msgr2=0x7f509c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 -- 172.21.15.89:0/1307167661 shutdown_connections 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.038+0000 7f50a3012700 1 -- 172.21.15.89:0/1307167661 wait complete. 2024-01-21T11:13:20.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a3012700 1 Processor -- start 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a3012700 1 -- start start 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a3012700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a3012700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0f4ff0 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a0dae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a0dae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41080/0 (socket says 172.21.15.89:41080) 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a0dae700 1 -- 172.21.15.89:0/207832927 learned_addr learned my addr 172.21.15.89:0/207832927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a0dae700 1 -- 172.21.15.89:0/207832927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090005040 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.039+0000 7f50a0dae700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5090013510 tx=0x7f509000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.040+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5090009320 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.040+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5090013f00 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.040+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c0f5190 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.040+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5090014dc0 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.040+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c0f5620 con 0x7f509c060280 2024-01-21T11:13:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.040+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50900263a0 con 0x7f509c060280 2024-01-21T11:13:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.041+0000 7f50997fa700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f508c02ffb0 0x7f508c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.041+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5090040fc0 con 0x7f509c060280 2024-01-21T11:13:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.041+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c059100 con 0x7f509c060280 2024-01-21T11:13:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.041+0000 7f509bfff700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f508c02ffb0 0x7f508c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.041+0000 7f509bfff700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f508c02ffb0 0x7f508c032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f50880096a0 tx=0x7f5088006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.045+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50900095f0 con 0x7f509c060280 2024-01-21T11:13:20.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.201+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f509c045ad0 con 0x7f509c060280 2024-01-21T11:13:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.204+0000 7f50997fa700 1 -- 172.21.15.89:0/207832927 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f509001db70 con 0x7f509c060280 2024-01-21T11:13:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f508c02ffb0 msgr2=0x7f508c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f508c02ffb0 0x7f508c032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f50880096a0 tx=0x7f5088006b40).stop 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 msgr2=0x7f509c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5090013510 tx=0x7f509000b040).stop 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 shutdown_connections 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f508c02ffb0 0x7f508c032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 --2- 172.21.15.89:0/207832927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 >> 172.21.15.89:0/207832927 conn(0x7f509c0f0b70 msgr2=0x7f509c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 shutdown_connections 2024-01-21T11:13:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:20.206+0000 7f50a3012700 1 -- 172.21.15.89:0/207832927 wait complete. 2024-01-21T11:13:20.573 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2024-01-21T11:13:20.574 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:13:20.574 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2024-01-21T11:13:20.595 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-01-21T11:13:20.595 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T11:13:20.660 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2024-01-21T11:13:20.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph orch host add smithi178 2024-01-21T11:13:20.882 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:19.651018+0000 mon.smithi089 (mon.0) 111 : cluster [DBG] mgrmap e16: smithi089.ufybqw(active, since 6s) 2024-01-21T11:13:20.882 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:20.205889+0000 mon.smithi089 (mon.0) 112 : audit [INF] from='client.? 172.21.15.89:0/207832927' entity='client.admin' 2024-01-21T11:13:22.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:21.433527+0000 mon.smithi089 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:22.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:21.688220+0000 mon.smithi089 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:23.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:23 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:21.694486+0000 mgr.smithi089.ufybqw (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi089 on smithi089 2024-01-21T11:13:34.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:33.640602+0000 mgr.smithi089.ufybqw (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:36.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:35.640856+0000 mgr.smithi089.ufybqw (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:38.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:37.641194+0000 mgr.smithi089.ufybqw (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:40.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:39.641450+0000 mgr.smithi089.ufybqw (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/2150498265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 msgr2=0x7fdfb40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 --2- 172.21.15.178:0/2150498265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb40f90f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fdfb0009fb0 tx=0x7fdfb0009b30).stop 2024-01-21T11:13:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/2150498265 shutdown_connections 2024-01-21T11:13:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 --2- 172.21.15.178:0/2150498265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb40f90f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/2150498265 >> 172.21.15.178:0/2150498265 conn(0x7fdfb40f0b70 msgr2=0x7fdfb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/2150498265 shutdown_connections 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.697+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/2150498265 wait complete. 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfbb5dc700 1 Processor -- start 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfbb5dc700 1 -- start start 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfbb5dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb41066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfbb5dc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4106ba0 con 0x7fdfb4060280 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfb9378700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb41066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfb9378700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb41066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34656/0 (socket says 172.21.15.178:34656) 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.698+0000 7fdfb9378700 1 -- 172.21.15.178:0/200846430 learned_addr learned my addr 172.21.15.178:0/200846430 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfb9378700 1 -- 172.21.15.178:0/200846430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0005040 con 0x7fdfb4060280 2024-01-21T11:13:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfb9378700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb41066b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdfb0013510 tx=0x7fdfb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:41.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfb0009320 con 0x7fdfb4060280 2024-01-21T11:13:41.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfb0013f00 con 0x7fdfb4060280 2024-01-21T11:13:41.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb4106da0 con 0x7fdfb4060280 2024-01-21T11:13:41.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfb0014dc0 con 0x7fdfb4060280 2024-01-21T11:13:41.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.699+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb4107230 con 0x7fdfb4060280 2024-01-21T11:13:41.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.700+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfb001d3d0 con 0x7fdfb4060280 2024-01-21T11:13:41.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.700+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb4045ad0 con 0x7fdfb4060280 2024-01-21T11:13:41.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.700+0000 7fdfa9ffb700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfa002ff60 0x7fdfa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:41.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.700+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfb0041400 con 0x7fdfb4060280 2024-01-21T11:13:41.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.700+0000 7fdfb8b77700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfa002ff60 0x7fdfa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:41.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.701+0000 7fdfb8b77700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfa002ff60 0x7fdfa0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdfa40096a0 tx=0x7fdfa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:41.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.705+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfb00126f0 con 0x7fdfb4060280 2024-01-21T11:13:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:41.875+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfb40f1d20 con 0x7fdfa002ff60 2024-01-21T11:13:42.470 INFO:teuthology.orchestra.run.smithi178.stdout:Added host 'smithi178' 2024-01-21T11:13:42.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.468+0000 7fdfa9ffb700 1 -- 172.21.15.178:0/200846430 <== mgr.14162 v2:172.21.15.89:6800/111466865 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fdfb40f1d20 con 0x7fdfa002ff60 2024-01-21T11:13:42.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfa002ff60 msgr2=0x7fdfa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:42.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfa002ff60 0x7fdfa0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdfa40096a0 tx=0x7fdfa4006b40).stop 2024-01-21T11:13:42.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 msgr2=0x7fdfb41066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb41066b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdfb0013510 tx=0x7fdfb000b040).stop 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 shutdown_connections 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfa002ff60 0x7fdfa0032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 --2- 172.21.15.178:0/200846430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb4060280 0x7fdfb41066b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 >> 172.21.15.178:0/200846430 conn(0x7fdfb40f0b70 msgr2=0x7fdfb40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 shutdown_connections 2024-01-21T11:13:42.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:42.470+0000 7fdfbb5dc700 1 -- 172.21.15.178:0/200846430 wait complete. 2024-01-21T11:13:42.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T11:13:42.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:41.641743+0000 mgr.smithi089.ufybqw (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:42.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:42.469153+0000 mon.smithi089 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:43.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:41.877880+0000 mgr.smithi089.ufybqw (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:43.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:42.469541+0000 mgr.smithi089.ufybqw (mgr.14162) 12 : cephadm [INF] Added host smithi178 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 -- 172.21.15.178:0/3540805463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 msgr2=0x7f6cf00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 --2- 172.21.15.178:0/3540805463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf00f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6ce4009fb0 tx=0x7f6ce4009b30).stop 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 -- 172.21.15.178:0/3540805463 shutdown_connections 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 --2- 172.21.15.178:0/3540805463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf00f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 -- 172.21.15.178:0/3540805463 >> 172.21.15.178:0/3540805463 conn(0x7f6cf00f1c70 msgr2=0x7f6cf00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 -- 172.21.15.178:0/3540805463 shutdown_connections 2024-01-21T11:13:43.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.943+0000 7f6cf5504700 1 -- 172.21.15.178:0/3540805463 wait complete. 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6cf5504700 1 Processor -- start 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6cf5504700 1 -- start start 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6cf5504700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6cf5504700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0106c20 con 0x7f6cf00f6510 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6ceeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6ceeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:39190/0 (socket says 172.21.15.178:39190) 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.944+0000 7f6ceeffd700 1 -- 172.21.15.178:0/2981545381 learned_addr learned my addr 172.21.15.178:0/2981545381 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6ceeffd700 1 -- 172.21.15.178:0/2981545381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4005040 con 0x7f6cf00f6510 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6ceeffd700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf0106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f6ce4013510 tx=0x7f6ce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:43.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ce4009320 con 0x7f6cf00f6510 2024-01-21T11:13:43.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ce4013f00 con 0x7f6cf00f6510 2024-01-21T11:13:43.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf0106e20 con 0x7f6cf00f6510 2024-01-21T11:13:43.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ce4014d90 con 0x7f6cf00f6510 2024-01-21T11:13:43.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.945+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf01072b0 con 0x7f6cf00f6510 2024-01-21T11:13:43.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.946+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ce4014420 con 0x7f6cf00f6510 2024-01-21T11:13:43.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.946+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf0045ad0 con 0x7f6cf00f6510 2024-01-21T11:13:43.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.946+0000 7f6cd77fe700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6cd802ff60 0x7f6cd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:43.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.946+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ce4041000 con 0x7f6cf00f6510 2024-01-21T11:13:43.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.946+0000 7f6cee7fc700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6cd802ff60 0x7f6cd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:43.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.946+0000 7f6cee7fc700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6cd802ff60 0x7f6cd8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f6ce00096a0 tx=0x7f6ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:43.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:43.950+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ce40095f0 con 0x7f6cf00f6510 2024-01-21T11:13:44.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.119+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6cf00f2e70 con 0x7f6cd802ff60 2024-01-21T11:13:44.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.121+0000 7f6cd77fe700 1 -- 172.21.15.178:0/2981545381 <== mgr.14162 v2:172.21.15.89:6800/111466865 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f6cf00f2e70 con 0x7f6cd802ff60 2024-01-21T11:13:44.122 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:13:44.122 INFO:teuthology.orchestra.run.smithi178.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "smithi178", "hostname": "smithi178", "labels": [], "status": ""}] 2024-01-21T11:13:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6cd802ff60 msgr2=0x7f6cd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6cd802ff60 0x7f6cd8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f6ce00096a0 tx=0x7f6ce0006b40).stop 2024-01-21T11:13:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 msgr2=0x7f6cf0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf0106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f6ce4013510 tx=0x7f6ce400b040).stop 2024-01-21T11:13:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 shutdown_connections 2024-01-21T11:13:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6cd802ff60 0x7f6cd8032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 --2- 172.21.15.178:0/2981545381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cf00f6510 0x7f6cf0106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 >> 172.21.15.178:0/2981545381 conn(0x7f6cf00f1c70 msgr2=0x7f6cf00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 shutdown_connections 2024-01-21T11:13:44.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:44.122+0000 7f6cf5504700 1 -- 172.21.15.178:0/2981545381 wait complete. 2024-01-21T11:13:44.531 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T11:13:44.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T11:13:44.835 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:43.641997+0000 mgr.smithi089.ufybqw (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:45.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 -- 172.21.15.89:0/671056761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 msgr2=0x7f36c40fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 --2- 172.21.15.89:0/671056761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c40fb570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f36ac009fb0 tx=0x7f36ac009b30).stop 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 -- 172.21.15.89:0/671056761 shutdown_connections 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 --2- 172.21.15.89:0/671056761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c40fb570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 -- 172.21.15.89:0/671056761 >> 172.21.15.89:0/671056761 conn(0x7f36c40f0b70 msgr2=0x7f36c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 -- 172.21.15.89:0/671056761 shutdown_connections 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.635+0000 7f36c9abf700 1 -- 172.21.15.89:0/671056761 wait complete. 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c9abf700 1 Processor -- start 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c9abf700 1 -- start start 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c9abf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c41088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c9abf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4108db0 con 0x7f36c4060200 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c41088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c41088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42164/0 (socket says 172.21.15.89:42164) 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c37fe700 1 -- 172.21.15.89:0/1230354539 learned_addr learned my addr 172.21.15.89:0/1230354539 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c37fe700 1 -- 172.21.15.89:0/1230354539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac005040 con 0x7f36c4060200 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.636+0000 7f36c37fe700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c41088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f36ac013510 tx=0x7f36ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.637+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36ac009320 con 0x7f36c4060200 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.637+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4108fb0 con 0x7f36c4060200 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.637+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36ac013f00 con 0x7f36c4060200 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.637+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36ac014dc0 con 0x7f36c4060200 2024-01-21T11:13:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.637+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4109440 con 0x7f36c4060200 2024-01-21T11:13:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.637+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36ac01d3d0 con 0x7f36c4060200 2024-01-21T11:13:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.638+0000 7f36bbfff700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36b002ff60 0x7f36b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.638+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36ac0410e0 con 0x7f36c4060200 2024-01-21T11:13:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.638+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c4045ad0 con 0x7f36c4060200 2024-01-21T11:13:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.638+0000 7f36c2ffd700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36b002ff60 0x7f36b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.638+0000 7f36c2ffd700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36b002ff60 0x7f36b0032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f36b40096a0 tx=0x7f36b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:45.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.642+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36ac012900 con 0x7f36c4060200 2024-01-21T11:13:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:45.797+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f36c410a190 con 0x7f36c4060200 2024-01-21T11:13:45.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:44.121855+0000 mgr.smithi089.ufybqw (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T11:13:46.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.663+0000 7f36bbfff700 1 -- 172.21.15.89:0/1230354539 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f36ac02b030 con 0x7f36c4060200 2024-01-21T11:13:46.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36b002ff60 msgr2=0x7f36b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:46.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36b002ff60 0x7f36b0032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f36b40096a0 tx=0x7f36b4006b40).stop 2024-01-21T11:13:46.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 msgr2=0x7f36c41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:46.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c41088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f36ac013510 tx=0x7f36ac00b040).stop 2024-01-21T11:13:46.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 shutdown_connections 2024-01-21T11:13:46.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36b002ff60 0x7f36b0032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:46.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 --2- 172.21.15.89:0/1230354539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c4060200 0x7f36c41088c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:46.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.665+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 >> 172.21.15.89:0/1230354539 conn(0x7f36c40f0b70 msgr2=0x7f36c40f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:46.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.666+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 shutdown_connections 2024-01-21T11:13:46.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-01-21T11:13:46.666+0000 7f36c9abf700 1 -- 172.21.15.89:0/1230354539 wait complete. 2024-01-21T11:13:46.667 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-01-21T11:13:46.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:45.642230+0000 mgr.smithi089.ufybqw (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:46.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:45.797995+0000 mon.smithi089 (mon.0) 116 : audit [INF] from='client.? 172.21.15.89:0/1230354539' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T11:13:47.079 INFO:tasks.cephadm:Adding mon.smithi089 on smithi089 2024-01-21T11:13:47.079 INFO:tasks.cephadm:Adding mon.smithi178 on smithi178 2024-01-21T11:13:47.079 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi089:172.21.15.89=smithi089;smithi178:172.21.15.178=smithi178' 2024-01-21T11:13:47.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:46.664048+0000 mon.smithi089 (mon.0) 117 : audit [INF] from='client.? 172.21.15.89:0/1230354539' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T11:13:47.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:46.664108+0000 mon.smithi089 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 -- 172.21.15.178:0/3501872408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 msgr2=0x7f56e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 --2- 172.21.15.178:0/3501872408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e40f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f56d8009fb0 tx=0x7f56d8009b30).stop 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 -- 172.21.15.178:0/3501872408 shutdown_connections 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 --2- 172.21.15.178:0/3501872408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e40f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 -- 172.21.15.178:0/3501872408 >> 172.21.15.178:0/3501872408 conn(0x7f56e40f09a0 msgr2=0x7f56e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 -- 172.21.15.178:0/3501872408 shutdown_connections 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 -- 172.21.15.178:0/3501872408 wait complete. 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.261+0000 7f56e3fff700 1 Processor -- start 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e3fff700 1 -- start start 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:48.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e3fff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4060220 con 0x7f56e40f51a0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:39216/0 (socket says 172.21.15.178:39216) 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e2ffd700 1 -- 172.21.15.178:0/4007034606 learned_addr learned my addr 172.21.15.178:0/4007034606 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e2ffd700 1 -- 172.21.15.178:0/4007034606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d8005040 con 0x7f56e40f51a0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.262+0000 7f56e2ffd700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e41091d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f56d8009830 tx=0x7f56d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d8009320 con 0x7f56e40f51a0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e4060420 con 0x7f56e40f51a0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56d80136c0 con 0x7f56e40f51a0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d801ee90 con 0x7f56e40f51a0 2024-01-21T11:13:48.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e4109880 con 0x7f56e40f51a0 2024-01-21T11:13:48.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56d801e420 con 0x7f56e40f51a0 2024-01-21T11:13:48.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.263+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e4059100 con 0x7f56e40f51a0 2024-01-21T11:13:48.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.264+0000 7f56cb7fe700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f56cc02ff10 0x7f56cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:48.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.264+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56d8041080 con 0x7f56e40f51a0 2024-01-21T11:13:48.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.264+0000 7f56e27fc700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f56cc02ff10 0x7f56cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:48.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.264+0000 7f56e27fc700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f56cc02ff10 0x7f56cc0323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f56d40096a0 tx=0x7f56d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:48.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.269+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56d806d2b0 con 0x7f56e40f51a0 2024-01-21T11:13:48.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.446+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 --> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi089:172.21.15.89=smithi089;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7f56e40f1c20 con 0x7f56cc02ff10 2024-01-21T11:13:48.452 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.451+0000 7f56cb7fe700 1 -- 172.21.15.178:0/4007034606 <== mgr.14162 v2:172.21.15.89:6800/111466865 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f56e40f1c20 con 0x7f56cc02ff10 2024-01-21T11:13:48.452 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f56cc02ff10 msgr2=0x7f56cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f56cc02ff10 0x7f56cc0323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f56d40096a0 tx=0x7f56d4006b40).stop 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 msgr2=0x7f56e41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e41091d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f56d8009830 tx=0x7f56d800b040).stop 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 shutdown_connections 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f56cc02ff10 0x7f56cc0323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 --2- 172.21.15.178:0/4007034606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e40f51a0 0x7f56e41091d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 >> 172.21.15.178:0/4007034606 conn(0x7f56e40f09a0 msgr2=0x7f56e40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 shutdown_connections 2024-01-21T11:13:48.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:48.453+0000 7f56e3fff700 1 -- 172.21.15.178:0/4007034606 wait complete. 2024-01-21T11:13:48.835 DEBUG:teuthology.orchestra.run.smithi178:mon.smithi178> sudo journalctl -f -n 0 -u ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178.service 2024-01-21T11:13:48.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:13:48.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:13:48.867 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:-- Logs begin at Sun 2024-01-21 10:54:43 UTC. -- 2024-01-21T11:13:48.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:47.642577+0000 mgr.smithi089.ufybqw (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:48.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:48.452571+0000 mon.smithi089 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:13:49.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:49 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:48.448457+0000 mgr.smithi089.ufybqw (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi089:172.21.15.89=smithi089;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:13:49.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:49 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:13:48.449917+0000 mgr.smithi089.ufybqw (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi089:172.21.15.89=smithi089;smithi178:172.21.15.178=smithi178;count:2 2024-01-21T11:13:49.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:49 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:49.642910+0000 mgr.smithi089.ufybqw (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:50.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 -- 172.21.15.178:0/4100404244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 msgr2=0x7facb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:50.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 --2- 172.21.15.178:0/4100404244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7facac009fb0 tx=0x7facac009b30).stop 2024-01-21T11:13:50.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 -- 172.21.15.178:0/4100404244 shutdown_connections 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 --2- 172.21.15.178:0/4100404244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 -- 172.21.15.178:0/4100404244 >> 172.21.15.178:0/4100404244 conn(0x7facb80f0b70 msgr2=0x7facb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 -- 172.21.15.178:0/4100404244 shutdown_connections 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 -- 172.21.15.178:0/4100404244 wait complete. 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 Processor -- start 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.044+0000 7facbf146700 1 -- start start 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbf146700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbf146700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb80fb820 con 0x7facb80f5410 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbcee2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbcee2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:39232/0 (socket says 172.21.15.178:39232) 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbcee2700 1 -- 172.21.15.178:0/3496671143 learned_addr learned my addr 172.21.15.178:0/3496671143 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbcee2700 1 -- 172.21.15.178:0/3496671143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac005040 con 0x7facb80f5410 2024-01-21T11:13:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbcee2700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80fd3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7facac009f80 tx=0x7facac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac009320 con 0x7facb80f5410 2024-01-21T11:13:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb80fb9c0 con 0x7facb80f5410 2024-01-21T11:13:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facac013ec0 con 0x7facb80f5410 2024-01-21T11:13:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.045+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac01ece0 con 0x7facb80f5410 2024-01-21T11:13:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.046+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb80fbe50 con 0x7facb80f5410 2024-01-21T11:13:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.046+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facac01e420 con 0x7facb80f5410 2024-01-21T11:13:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.046+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb8059100 con 0x7facb80f5410 2024-01-21T11:13:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.046+0000 7facb57fa700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faca802ff60 0x7faca8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.046+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facac040e30 con 0x7facb80f5410 2024-01-21T11:13:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.047+0000 7facb7fff700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faca802ff60 0x7faca8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.047+0000 7facb7fff700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faca802ff60 0x7faca8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7faca40096a0 tx=0x7faca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:50.051 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.051+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facac01e730 con 0x7facb80f5410 2024-01-21T11:13:50.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.254+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facb80fae60 con 0x7facb80f5410 2024-01-21T11:13:50.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.255+0000 7facb57fa700 1 -- 172.21.15.178:0/3496671143 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facac02c2d0 con 0x7facb80f5410 2024-01-21T11:13:50.255 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:13:50.255 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:13:50.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faca802ff60 msgr2=0x7faca8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:50.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faca802ff60 0x7faca8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7faca40096a0 tx=0x7faca4006b40).stop 2024-01-21T11:13:50.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 msgr2=0x7facb80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:50.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80fd3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7facac009f80 tx=0x7facac00b040).stop 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 shutdown_connections 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faca802ff60 0x7faca8032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 --2- 172.21.15.178:0/3496671143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80f5410 0x7facb80fd3a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 >> 172.21.15.178:0/3496671143 conn(0x7facb80f0b70 msgr2=0x7facb805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 shutdown_connections 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:50.256+0000 7facbf146700 1 -- 172.21.15.178:0/3496671143 wait complete. 2024-01-21T11:13:50.257 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:13:50.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:50.255542+0000 mon.smithi089 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.178:0/3496671143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:13:51.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:13:51.635 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:13:51.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:51 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:51.643200+0000 mgr.smithi089.ufybqw (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:53.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- 172.21.15.178:0/2402367685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 msgr2=0x7f4b080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:53.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 --2- 172.21.15.178:0/2402367685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b080f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f4afc009fb0 tx=0x7f4afc009b30).stop 2024-01-21T11:13:53.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- 172.21.15.178:0/2402367685 shutdown_connections 2024-01-21T11:13:53.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 --2- 172.21.15.178:0/2402367685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b080f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:53.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- 172.21.15.178:0/2402367685 >> 172.21.15.178:0/2402367685 conn(0x7f4b080f09a0 msgr2=0x7f4b080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:53.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- 172.21.15.178:0/2402367685 shutdown_connections 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- 172.21.15.178:0/2402367685 wait complete. 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 Processor -- start 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- start start 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b081053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.147+0000 7f4b0759e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b081058a0 con 0x7f4b080f51a0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b081053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b081053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40852/0 (socket says 172.21.15.178:40852) 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0659c700 1 -- 172.21.15.178:0/2457044385 learned_addr learned my addr 172.21.15.178:0/2457044385 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0659c700 1 -- 172.21.15.178:0/2457044385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc005040 con 0x7f4b080f51a0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0659c700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b081053b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4afc009830 tx=0x7f4afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4afc009320 con 0x7f4b080f51a0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b08105aa0 con 0x7f4b080f51a0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4afc0136c0 con 0x7f4b080f51a0 2024-01-21T11:13:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4afc01ee90 con 0x7f4b080f51a0 2024-01-21T11:13:53.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.148+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b08105f30 con 0x7f4b080f51a0 2024-01-21T11:13:53.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.149+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4afc01e420 con 0x7f4b080f51a0 2024-01-21T11:13:53.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.149+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08045ad0 con 0x7f4b080f51a0 2024-01-21T11:13:53.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.149+0000 7f4af6ffd700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4af002ff10 0x7f4af00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:53.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.149+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4afc040fc0 con 0x7f4b080f51a0 2024-01-21T11:13:53.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.150+0000 7f4b05d9b700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4af002ff10 0x7f4af00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:53.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.150+0000 7f4b05d9b700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4af002ff10 0x7f4af00323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f4af80096a0 tx=0x7f4af8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:53.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.154+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4afc06d2b0 con 0x7f4b080f51a0 2024-01-21T11:13:53.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.358+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b08106de0 con 0x7f4b080f51a0 2024-01-21T11:13:53.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.359+0000 7f4af6ffd700 1 -- 172.21.15.178:0/2457044385 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4afc02c2d0 con 0x7f4b080f51a0 2024-01-21T11:13:53.359 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:13:53.359 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:13:53.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4af002ff10 msgr2=0x7f4af00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:53.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4af002ff10 0x7f4af00323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f4af80096a0 tx=0x7f4af8006b40).stop 2024-01-21T11:13:53.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 msgr2=0x7f4b081053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:53.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b081053b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4afc009830 tx=0x7f4afc00b040).stop 2024-01-21T11:13:53.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 shutdown_connections 2024-01-21T11:13:53.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4af002ff10 0x7f4af00323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:53.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 --2- 172.21.15.178:0/2457044385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b080f51a0 0x7f4b081053b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:53.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 >> 172.21.15.178:0/2457044385 conn(0x7f4b080f09a0 msgr2=0x7f4b080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:53.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 shutdown_connections 2024-01-21T11:13:53.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:53.361+0000 7f4b0759e700 1 -- 172.21.15.178:0/2457044385 wait complete. 2024-01-21T11:13:53.362 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:13:53.683 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:53.359827+0000 mon.smithi089 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.178:0/2457044385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:13:54.684 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:53.643468+0000 mgr.smithi089.ufybqw (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:54.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:13:54.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 -- 172.21.15.178:0/3396485085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 msgr2=0x7fa5f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 --2- 172.21.15.178:0/3396485085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f90f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa5e8009fb0 tx=0x7fa5e8009b30).stop 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 -- 172.21.15.178:0/3396485085 shutdown_connections 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 --2- 172.21.15.178:0/3396485085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f90f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 -- 172.21.15.178:0/3396485085 >> 172.21.15.178:0/3396485085 conn(0x7fa5f80f0b70 msgr2=0x7fa5f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 -- 172.21.15.178:0/3396485085 shutdown_connections 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.950+0000 7fa5fce5c700 1 -- 172.21.15.178:0/3396485085 wait complete. 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5fce5c700 1 Processor -- start 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5fce5c700 1 -- start start 2024-01-21T11:13:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5fce5c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5fce5c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80f4ff0 con 0x7fa5f8060280 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5f659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5f659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40870/0 (socket says 172.21.15.178:40870) 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.951+0000 7fa5f659c700 1 -- 172.21.15.178:0/1535953005 learned_addr learned my addr 172.21.15.178:0/1535953005 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5f659c700 1 -- 172.21.15.178:0/1535953005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8005040 con 0x7fa5f8060280 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5f659c700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f6b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa5e8009830 tx=0x7fa5e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e8009320 con 0x7fa5f8060280 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f80f5190 con 0x7fa5f8060280 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5e80136c0 con 0x7fa5f8060280 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e801ee90 con 0x7fa5f8060280 2024-01-21T11:13:55.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.952+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f80f5620 con 0x7fa5f8060280 2024-01-21T11:13:55.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.953+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5e801e420 con 0x7fa5f8060280 2024-01-21T11:13:55.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.953+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f8059100 con 0x7fa5f8060280 2024-01-21T11:13:55.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.953+0000 7fa5e6ffd700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa5e002ff10 0x7fa5e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:55.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.953+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5e8040f40 con 0x7fa5f8060280 2024-01-21T11:13:55.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.953+0000 7fa5f5d9b700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa5e002ff10 0x7fa5e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:55.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.953+0000 7fa5f5d9b700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa5e002ff10 0x7fa5e00323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa5ec0096a0 tx=0x7fa5ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:55.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:55.957+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5e806d2b0 con 0x7fa5f8060280 2024-01-21T11:13:56.161 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.161+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5f80fbf50 con 0x7fa5f8060280 2024-01-21T11:13:56.162 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.162+0000 7fa5e6ffd700 1 -- 172.21.15.178:0/1535953005 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa5e80095f0 con 0x7fa5f8060280 2024-01-21T11:13:56.162 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:13:56.162 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.163+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa5e002ff10 msgr2=0x7fa5e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.163+0000 7fa5fce5c700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa5e002ff10 0x7fa5e00323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa5ec0096a0 tx=0x7fa5ec006b40).stop 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.163+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 msgr2=0x7fa5f80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.163+0000 7fa5fce5c700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f6b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa5e8009830 tx=0x7fa5e800b040).stop 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.164+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 shutdown_connections 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.164+0000 7fa5fce5c700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa5e002ff10 0x7fa5e00323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.164+0000 7fa5fce5c700 1 --2- 172.21.15.178:0/1535953005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5f8060280 0x7fa5f80f6b70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.164+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 >> 172.21.15.178:0/1535953005 conn(0x7fa5f80f0b70 msgr2=0x7fa5f80f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.164+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 shutdown_connections 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:56.164+0000 7fa5fce5c700 1 -- 172.21.15.178:0/1535953005 wait complete. 2024-01-21T11:13:56.164 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:13:56.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:55.643803+0000 mgr.smithi089.ufybqw (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:56.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:56.162691+0000 mon.smithi089 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.178:0/1535953005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:13:57.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:13:57.549 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 -- 172.21.15.178:0/3365379275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 msgr2=0x7facd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 --2- 172.21.15.178:0/3365379275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd00f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7facc0009fb0 tx=0x7facc0009b30).stop 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 -- 172.21.15.178:0/3365379275 shutdown_connections 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 --2- 172.21.15.178:0/3365379275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd00f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 -- 172.21.15.178:0/3365379275 >> 172.21.15.178:0/3365379275 conn(0x7facd00f0b70 msgr2=0x7facd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 -- 172.21.15.178:0/3365379275 shutdown_connections 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.703+0000 7facd79ca700 1 -- 172.21.15.178:0/3365379275 wait complete. 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd79ca700 1 Processor -- start 2024-01-21T11:13:58.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd79ca700 1 -- start start 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd79ca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd79ca700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0106c00 con 0x7facd00f5410 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd5766700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd5766700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40886/0 (socket says 172.21.15.178:40886) 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.704+0000 7facd5766700 1 -- 172.21.15.178:0/1436229073 learned_addr learned my addr 172.21.15.178:0/1436229073 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facd5766700 1 -- 172.21.15.178:0/1436229073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0005040 con 0x7facd00f5410 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facd5766700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd0106710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7facc0014000 tx=0x7facc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facc00095f0 con 0x7facd00f5410 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd0106e00 con 0x7facd00f5410 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facc0007e70 con 0x7facd00f5410 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facc0014ea0 con 0x7facd00f5410 2024-01-21T11:13:58.706 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.705+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd0107290 con 0x7facd00f5410 2024-01-21T11:13:58.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.706+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facc0025070 con 0x7facd00f5410 2024-01-21T11:13:58.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.706+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd0045ad0 con 0x7facd00f5410 2024-01-21T11:13:58.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.706+0000 7facc5ffb700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facbc02ff60 0x7facbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:13:58.708 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.706+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facc0040a40 con 0x7facd00f5410 2024-01-21T11:13:58.708 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.706+0000 7facd4f65700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facbc02ff60 0x7facbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:13:58.708 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.707+0000 7facd4f65700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facbc02ff60 0x7facbc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7faccc0096a0 tx=0x7faccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:13:58.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.711+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facc00079c0 con 0x7facd00f5410 2024-01-21T11:13:58.915 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.915+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facd0108140 con 0x7facd00f5410 2024-01-21T11:13:58.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.916+0000 7facc5ffb700 1 -- 172.21.15.178:0/1436229073 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facc000f300 con 0x7facd00f5410 2024-01-21T11:13:58.916 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:13:58.916 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facbc02ff60 msgr2=0x7facbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facbc02ff60 0x7facbc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7faccc0096a0 tx=0x7faccc006b40).stop 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 msgr2=0x7facd0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd0106710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7facc0014000 tx=0x7facc000b040).stop 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 shutdown_connections 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facbc02ff60 0x7facbc032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 --2- 172.21.15.178:0/1436229073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facd00f5410 0x7facd0106710 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:13:58.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 >> 172.21.15.178:0/1436229073 conn(0x7facd00f0b70 msgr2=0x7facd005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:13:58.919 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 shutdown_connections 2024-01-21T11:13:58.919 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:13:58.918+0000 7facd79ca700 1 -- 172.21.15.178:0/1436229073 wait complete. 2024-01-21T11:13:58.919 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:13:58.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:57.644081+0000 mgr.smithi089.ufybqw (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:13:59.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:13:59 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:13:58.916887+0000 mon.smithi089 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.178:0/1436229073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:00.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:00.314 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:00.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:13:59.644367+0000 mgr.smithi089.ufybqw (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 -- 172.21.15.178:0/3918028334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 msgr2=0x7fb0040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 --2- 172.21.15.178:0/3918028334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0040f90f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7faff8009fb0 tx=0x7faff8009b30).stop 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 -- 172.21.15.178:0/3918028334 shutdown_connections 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 --2- 172.21.15.178:0/3918028334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0040f90f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 -- 172.21.15.178:0/3918028334 >> 172.21.15.178:0/3918028334 conn(0x7fb0040f0b70 msgr2=0x7fb0040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 -- 172.21.15.178:0/3918028334 shutdown_connections 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.443+0000 7fb00acb0700 1 -- 172.21.15.178:0/3918028334 wait complete. 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb00acb0700 1 Processor -- start 2024-01-21T11:14:01.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb00acb0700 1 -- start start 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb00acb0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0041066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb00acb0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004106bd0 con 0x7fb004060280 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb008a4c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0041066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb008a4c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0041066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40904/0 (socket says 172.21.15.178:40904) 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.444+0000 7fb008a4c700 1 -- 172.21.15.178:0/2749018448 learned_addr learned my addr 172.21.15.178:0/2749018448 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb008a4c700 1 -- 172.21.15.178:0/2749018448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff8005040 con 0x7fb004060280 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb008a4c700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0041066e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7faff8013510 tx=0x7faff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff8009320 con 0x7fb004060280 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb004106dd0 con 0x7fb004060280 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faff8013f00 con 0x7fb004060280 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff8014dc0 con 0x7fb004060280 2024-01-21T11:14:01.446 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.445+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004107260 con 0x7fb004060280 2024-01-21T11:14:01.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.446+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faff8014420 con 0x7fb004060280 2024-01-21T11:14:01.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.446+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb004045ad0 con 0x7fb004060280 2024-01-21T11:14:01.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.446+0000 7fb0017fa700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faff402ff60 0x7faff4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:01.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.446+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faff8041110 con 0x7fb004060280 2024-01-21T11:14:01.447 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.446+0000 7fb003fff700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faff402ff60 0x7faff4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:01.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.447+0000 7fb003fff700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faff402ff60 0x7faff4032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7faff00096a0 tx=0x7faff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.451+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faff80095f0 con 0x7fb004060280 2024-01-21T11:14:01.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.657+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb004108110 con 0x7fb004060280 2024-01-21T11:14:01.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.658+0000 7fb0017fa700 1 -- 172.21.15.178:0/2749018448 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faff801cb10 con 0x7fb004060280 2024-01-21T11:14:01.658 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:01.658 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faff402ff60 msgr2=0x7faff4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faff402ff60 0x7faff4032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7faff00096a0 tx=0x7faff0006b40).stop 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 msgr2=0x7fb0041066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0041066e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7faff8013510 tx=0x7faff800b040).stop 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 shutdown_connections 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7faff402ff60 0x7faff4032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 --2- 172.21.15.178:0/2749018448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb004060280 0x7fb0041066e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 >> 172.21.15.178:0/2749018448 conn(0x7fb0040f0b70 msgr2=0x7fb0040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 shutdown_connections 2024-01-21T11:14:01.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:01.660+0000 7fb00acb0700 1 -- 172.21.15.178:0/2749018448 wait complete. 2024-01-21T11:14:01.661 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:02.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:01.644722+0000 mgr.smithi089.ufybqw (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:02.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:01.658817+0000 mon.smithi089 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.178:0/2749018448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:03.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:03.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:04.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:03.645065+0000 mgr.smithi089.ufybqw (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:06.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:05.645426+0000 mgr.smithi089.ufybqw (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 -- 172.21.15.178:0/907389249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 msgr2=0x7fdde40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 --2- 172.21.15.178:0/907389249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde40f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fddd4009fb0 tx=0x7fddd4009b30).stop 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 -- 172.21.15.178:0/907389249 shutdown_connections 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 --2- 172.21.15.178:0/907389249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde40f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 -- 172.21.15.178:0/907389249 >> 172.21.15.178:0/907389249 conn(0x7fdde40f1c70 msgr2=0x7fdde40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 -- 172.21.15.178:0/907389249 shutdown_connections 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.954+0000 7fdde92a0700 1 -- 172.21.15.178:0/907389249 wait complete. 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde92a0700 1 Processor -- start 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde92a0700 1 -- start start 2024-01-21T11:14:08.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde92a0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde92a0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4106c20 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:37688/0 (socket says 172.21.15.178:37688) 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.955+0000 7fdde2ffd700 1 -- 172.21.15.178:0/2868487516 learned_addr learned my addr 172.21.15.178:0/2868487516 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fdde2ffd700 1 -- 172.21.15.178:0/2868487516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4005040 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fdde2ffd700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde4106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fddd4014000 tx=0x7fddd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddd40095f0 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4106e20 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddd4007e70 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddd4014ea0 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.956+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde41072b0 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.957+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddd401d4e0 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.957+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4045ad0 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.957+0000 7fddcb7fe700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fddcc02ff60 0x7fddcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.957+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddd4040c80 con 0x7fdde40f6510 2024-01-21T11:14:08.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.957+0000 7fdde27fc700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fddcc02ff60 0x7fddcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.957+0000 7fdde27fc700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fddcc02ff60 0x7fddcc032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fddd80096a0 tx=0x7fddd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:08.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:07.962+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddd4025e60 con 0x7fdde40f6510 2024-01-21T11:14:08.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.170+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdde40fbf60 con 0x7fdde40f6510 2024-01-21T11:14:08.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.170+0000 7fddcb7fe700 1 -- 172.21.15.178:0/2868487516 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fddd4012330 con 0x7fdde40f6510 2024-01-21T11:14:08.170 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:08.171 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fddcc02ff60 msgr2=0x7fddcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fddcc02ff60 0x7fddcc032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fddd80096a0 tx=0x7fddd8006b40).stop 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 msgr2=0x7fdde4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde4106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fddd4014000 tx=0x7fddd400b040).stop 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 shutdown_connections 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fddcc02ff60 0x7fddcc032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 --2- 172.21.15.178:0/2868487516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde40f6510 0x7fdde4106730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 >> 172.21.15.178:0/2868487516 conn(0x7fdde40f1c70 msgr2=0x7fdde40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:08.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 shutdown_connections 2024-01-21T11:14:08.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:08.172+0000 7fdde92a0700 1 -- 172.21.15.178:0/2868487516 wait complete. 2024-01-21T11:14:08.173 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:08.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:07.645739+0000 mgr.smithi089.ufybqw (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:08.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:08.171196+0000 mon.smithi089 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.178:0/2868487516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:10.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:09.646008+0000 mgr.smithi089.ufybqw (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:11.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:11.725 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:12.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:11.646348+0000 mgr.smithi089.ufybqw (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.172+0000 7fc2834fd700 1 -- 172.21.15.178:0/4088593105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 msgr2=0x7fc27c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.172+0000 7fc2834fd700 1 --2- 172.21.15.178:0/4088593105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c0f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fc26c009fb0 tx=0x7fc26c009b30).stop 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.172+0000 7fc2834fd700 1 -- 172.21.15.178:0/4088593105 shutdown_connections 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.172+0000 7fc2834fd700 1 --2- 172.21.15.178:0/4088593105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c0f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.172+0000 7fc2834fd700 1 -- 172.21.15.178:0/4088593105 >> 172.21.15.178:0/4088593105 conn(0x7fc27c0f0b70 msgr2=0x7fc27c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc2834fd700 1 -- 172.21.15.178:0/4088593105 shutdown_connections 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc2834fd700 1 -- 172.21.15.178:0/4088593105 wait complete. 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc2834fd700 1 Processor -- start 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc2834fd700 1 -- start start 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc2834fd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc2834fd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c060220 con 0x7fc27c0f5410 2024-01-21T11:14:14.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc281299700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc281299700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43642/0 (socket says 172.21.15.178:43642) 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.173+0000 7fc281299700 1 -- 172.21.15.178:0/2921978899 learned_addr learned my addr 172.21.15.178:0/2921978899 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc281299700 1 -- 172.21.15.178:0/2921978899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c005040 con 0x7fc27c0f5410 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc281299700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc26c013510 tx=0x7fc26c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26c009320 con 0x7fc27c0f5410 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27c060420 con 0x7fc27c0f5410 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc26c013f00 con 0x7fc27c0f5410 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26c014d90 con 0x7fc27c0f5410 2024-01-21T11:14:14.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.174+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27c109b00 con 0x7fc27c0f5410 2024-01-21T11:14:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.175+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27c059100 con 0x7fc27c0f5410 2024-01-21T11:14:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.175+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc26c0095f0 con 0x7fc27c0f5410 2024-01-21T11:14:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.175+0000 7fc271ffb700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc26802ff60 0x7fc268032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.175+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc26c040fc0 con 0x7fc27c0f5410 2024-01-21T11:14:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.175+0000 7fc280a98700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc26802ff60 0x7fc268032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.176+0000 7fc280a98700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc26802ff60 0x7fc268032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc2780096a0 tx=0x7fc278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:14.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.180+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc26c014420 con 0x7fc27c0f5410 2024-01-21T11:14:14.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.383+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc27c045ad0 con 0x7fc27c0f5410 2024-01-21T11:14:14.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.384+0000 7fc271ffb700 1 -- 172.21.15.178:0/2921978899 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc26c010230 con 0x7fc27c0f5410 2024-01-21T11:14:14.384 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:14.384 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc26802ff60 msgr2=0x7fc268032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc26802ff60 0x7fc268032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc2780096a0 tx=0x7fc278006b40).stop 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 msgr2=0x7fc27c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc26c013510 tx=0x7fc26c00b040).stop 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 shutdown_connections 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc26802ff60 0x7fc268032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 --2- 172.21.15.178:0/2921978899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc27c0f5410 0x7fc27c109450 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 >> 172.21.15.178:0/2921978899 conn(0x7fc27c0f0b70 msgr2=0x7fc27c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 shutdown_connections 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:14.386+0000 7fc2834fd700 1 -- 172.21.15.178:0/2921978899 wait complete. 2024-01-21T11:14:14.386 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:13.646709+0000 mgr.smithi089.ufybqw (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:13.771433+0000 mon.smithi089 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:14:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:13.780751+0000 mon.smithi089 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:14:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:14.384747+0000 mon.smithi089 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.178:0/2921978899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:15.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:15.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:16.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:15.647082+0000 mgr.smithi089.ufybqw (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:16.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 -- 172.21.15.178:0/1348944267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 msgr2=0x7f6ffc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:16.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 --2- 172.21.15.178:0/1348944267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc0f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6fec009fb0 tx=0x7f6fec009b30).stop 2024-01-21T11:14:16.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 -- 172.21.15.178:0/1348944267 shutdown_connections 2024-01-21T11:14:16.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 --2- 172.21.15.178:0/1348944267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc0f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 -- 172.21.15.178:0/1348944267 >> 172.21.15.178:0/1348944267 conn(0x7f6ffc0f1c70 msgr2=0x7f6ffc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 -- 172.21.15.178:0/1348944267 shutdown_connections 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.953+0000 7f70013d5700 1 -- 172.21.15.178:0/1348944267 wait complete. 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f70013d5700 1 Processor -- start 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f70013d5700 1 -- start start 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f70013d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f70013d5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc106bc0 con 0x7f6ffc0f6510 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f6ffaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f6ffaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43656/0 (socket says 172.21.15.178:43656) 2024-01-21T11:14:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.954+0000 7f6ffaffd700 1 -- 172.21.15.178:0/334144378 learned_addr learned my addr 172.21.15.178:0/334144378 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f6ffaffd700 1 -- 172.21.15.178:0/334144378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec005040 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f6ffaffd700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc1066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6fec014000 tx=0x7f6fec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec0095f0 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc106dc0 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fec007e70 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec014ea0 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.955+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc107250 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.956+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fec025070 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.956+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc045ad0 con 0x7f6ffc0f6510 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.956+0000 7f6fe37fe700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fe402ff60 0x7f6fe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:16.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.956+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fec040a00 con 0x7f6ffc0f6510 2024-01-21T11:14:16.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.956+0000 7f6ffa7fc700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fe402ff60 0x7f6fe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:16.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.957+0000 7f6ffa7fc700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fe402ff60 0x7f6fe4032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f6ff00096a0 tx=0x7f6ff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:16.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:16.961+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fec0079c0 con 0x7f6ffc0f6510 2024-01-21T11:14:17.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.170+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ffc1080a0 con 0x7f6ffc0f6510 2024-01-21T11:14:17.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.171+0000 7f6fe37fe700 1 -- 172.21.15.178:0/334144378 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fec00f300 con 0x7f6ffc0f6510 2024-01-21T11:14:17.171 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:17.171 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:17.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fe402ff60 msgr2=0x7f6fe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:17.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fe402ff60 0x7f6fe4032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f6ff00096a0 tx=0x7f6ff0006b40).stop 2024-01-21T11:14:17.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 msgr2=0x7f6ffc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:17.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc1066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6fec014000 tx=0x7f6fec00b040).stop 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 shutdown_connections 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fe402ff60 0x7f6fe4032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 --2- 172.21.15.178:0/334144378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ffc0f6510 0x7f6ffc1066d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 >> 172.21.15.178:0/334144378 conn(0x7f6ffc0f1c70 msgr2=0x7f6ffc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 shutdown_connections 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:17.172+0000 7f70013d5700 1 -- 172.21.15.178:0/334144378 wait complete. 2024-01-21T11:14:17.173 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:17.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:17.171365+0000 mon.smithi089 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.178:0/334144378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:18.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:18.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:18.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:17.647438+0000 mgr.smithi089.ufybqw (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:19.718 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1394417025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 msgr2=0x7ffb840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:19.718 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 --2- 172.21.15.178:0/1394417025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb840f55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ffb74009fb0 tx=0x7ffb74009b30).stop 2024-01-21T11:14:19.718 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1394417025 shutdown_connections 2024-01-21T11:14:19.718 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 --2- 172.21.15.178:0/1394417025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb840f55c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1394417025 >> 172.21.15.178:0/1394417025 conn(0x7ffb840f09a0 msgr2=0x7ffb840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1394417025 shutdown_connections 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1394417025 wait complete. 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 Processor -- start 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- start start 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb841064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.716+0000 7ffb8b0e9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb841069b0 con 0x7ffb840f51a0 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8a0e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb841064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8a0e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb841064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43678/0 (socket says 172.21.15.178:43678) 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8a0e7700 1 -- 172.21.15.178:0/1345388884 learned_addr learned my addr 172.21.15.178:0/1345388884 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8a0e7700 1 -- 172.21.15.178:0/1345388884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb74005040 con 0x7ffb840f51a0 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8a0e7700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb841064c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ffb74009830 tx=0x7ffb7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:19.719 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb74009320 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb740136c0 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb84106bb0 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb7401ee90 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.717+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb84107040 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.718+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb74014440 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.718+0000 7ffb7affd700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ffb7002ff10 0x7ffb700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.718+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb74040fb0 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.718+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb84045ad0 con 0x7ffb840f51a0 2024-01-21T11:14:19.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.718+0000 7ffb898e6700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ffb7002ff10 0x7ffb700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:19.721 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.719+0000 7ffb898e6700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ffb7002ff10 0x7ffb700323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ffb800096a0 tx=0x7ffb80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:19.722 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.723+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb740273f0 con 0x7ffb840f51a0 2024-01-21T11:14:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.930+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb840facd0 con 0x7ffb840f51a0 2024-01-21T11:14:19.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.931+0000 7ffb7affd700 1 -- 172.21.15.178:0/1345388884 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb7401c320 con 0x7ffb840f51a0 2024-01-21T11:14:19.931 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:19.931 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.932+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ffb7002ff10 msgr2=0x7ffb700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.932+0000 7ffb8b0e9700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ffb7002ff10 0x7ffb700323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ffb800096a0 tx=0x7ffb80006b40).stop 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.932+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 msgr2=0x7ffb841064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.932+0000 7ffb8b0e9700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb841064c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ffb74009830 tx=0x7ffb7400b040).stop 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.933+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 shutdown_connections 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.933+0000 7ffb8b0e9700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ffb7002ff10 0x7ffb700323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.933+0000 7ffb8b0e9700 1 --2- 172.21.15.178:0/1345388884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb840f51a0 0x7ffb841064c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.933+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 >> 172.21.15.178:0/1345388884 conn(0x7ffb840f09a0 msgr2=0x7ffb840f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.933+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 shutdown_connections 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:19.933+0000 7ffb8b0e9700 1 -- 172.21.15.178:0/1345388884 wait complete. 2024-01-21T11:14:19.933 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:20.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:19.647747+0000 mgr.smithi089.ufybqw (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:20.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:19.931345+0000 mon.smithi089 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.178:0/1345388884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:21.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:21.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:22.469 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3276872765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 msgr2=0x7f1f040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 --2- 172.21.15.178:0/3276872765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f040f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f1eec009fb0 tx=0x7f1eec009b30).stop 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3276872765 shutdown_connections 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 --2- 172.21.15.178:0/3276872765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f040f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3276872765 >> 172.21.15.178:0/3276872765 conn(0x7f1f040f0b70 msgr2=0x7f1f040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3276872765 shutdown_connections 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.465+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3276872765 wait complete. 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f08bb1700 1 Processor -- start 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f08bb1700 1 -- start start 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f08bb1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f08bb1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f04060220 con 0x7f1f040f5410 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f0259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:22.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f0259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56876/0 (socket says 172.21.15.178:56876) 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f0259c700 1 -- 172.21.15.178:0/3938272794 learned_addr learned my addr 172.21.15.178:0/3938272794 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.466+0000 7f1f0259c700 1 -- 172.21.15.178:0/3938272794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec005040 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.467+0000 7f1f0259c700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04109450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1eec013510 tx=0x7f1eec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.467+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eec009320 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.467+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1eec013f00 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.467+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f04060420 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.467+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eec014d90 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.467+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f04109b00 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.468+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1eec0263a0 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.468+0000 7f1efaffd700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ff60 0x7f1ef0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.468+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1eec0410f0 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.468+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04059100 con 0x7f1f040f5410 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.468+0000 7f1f01d9b700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ff60 0x7f1ef0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:22.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.468+0000 7f1f01d9b700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ff60 0x7f1ef0032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1ef40096a0 tx=0x7f1ef4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:22.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.472+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1eec012920 con 0x7f1f040f5410 2024-01-21T11:14:22.679 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.678+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f0410a1e0 con 0x7f1f040f5410 2024-01-21T11:14:22.680 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.679+0000 7f1efaffd700 1 -- 172.21.15.178:0/3938272794 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1eec02b020 con 0x7f1f040f5410 2024-01-21T11:14:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:22.680 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:22.681 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ff60 msgr2=0x7f1ef0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:22.681 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ff60 0x7f1ef0032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1ef40096a0 tx=0x7f1ef4006b40).stop 2024-01-21T11:14:22.681 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 msgr2=0x7f1f04109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:22.681 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04109450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1eec013510 tx=0x7f1eec00b040).stop 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 shutdown_connections 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ff60 0x7f1ef0032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 --2- 172.21.15.178:0/3938272794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04109450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 >> 172.21.15.178:0/3938272794 conn(0x7f1f040f0b70 msgr2=0x7f1f040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 shutdown_connections 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:22.680+0000 7f1f08bb1700 1 -- 172.21.15.178:0/3938272794 wait complete. 2024-01-21T11:14:22.682 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:22.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:21.648032+0000 mgr.smithi089.ufybqw (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:23.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:23 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:22.680538+0000 mon.smithi089 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.178:0/3938272794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:24.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:24.079 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:24.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:23.648380+0000 mgr.smithi089.ufybqw (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:25.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.263+0000 7fc2da47d700 1 -- 172.21.15.178:0/1360422205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 msgr2=0x7fc2d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:25.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.263+0000 7fc2da47d700 1 --2- 172.21.15.178:0/1360422205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d40f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fc2c4009fb0 tx=0x7fc2c4009b30).stop 2024-01-21T11:14:25.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.263+0000 7fc2da47d700 1 -- 172.21.15.178:0/1360422205 shutdown_connections 2024-01-21T11:14:25.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.263+0000 7fc2da47d700 1 --2- 172.21.15.178:0/1360422205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d40f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:25.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.263+0000 7fc2da47d700 1 -- 172.21.15.178:0/1360422205 >> 172.21.15.178:0/1360422205 conn(0x7fc2d40f0b70 msgr2=0x7fc2d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.263+0000 7fc2da47d700 1 -- 172.21.15.178:0/1360422205 shutdown_connections 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.264+0000 7fc2da47d700 1 -- 172.21.15.178:0/1360422205 wait complete. 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.264+0000 7fc2da47d700 1 Processor -- start 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.264+0000 7fc2da47d700 1 -- start start 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.264+0000 7fc2da47d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.264+0000 7fc2da47d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4060240 con 0x7fc2d40f5410 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56902/0 (socket says 172.21.15.178:56902) 2024-01-21T11:14:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d3fff700 1 -- 172.21.15.178:0/421385071 learned_addr learned my addr 172.21.15.178:0/421385071 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d3fff700 1 -- 172.21.15.178:0/421385071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4005040 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d3fff700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d4109430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc2c4013510 tx=0x7fc2c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c4009320 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d4060440 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2c4013f00 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c4014d90 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.265+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d4109ae0 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.266+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2c40263a0 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.266+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d4045ad0 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.266+0000 7fc2d0ff9700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc2bc02ff60 0x7fc2bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.266+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2c40413b0 con 0x7fc2d40f5410 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.266+0000 7fc2d37fe700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc2bc02ff60 0x7fc2bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:25.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.267+0000 7fc2d37fe700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc2bc02ff60 0x7fc2bc032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc2c80096a0 tx=0x7fc2c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:25.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.271+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2c4012720 con 0x7fc2d40f5410 2024-01-21T11:14:25.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.474+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2d40fae60 con 0x7fc2d40f5410 2024-01-21T11:14:25.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.474+0000 7fc2d0ff9700 1 -- 172.21.15.178:0/421385071 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2c4012330 con 0x7fc2d40f5410 2024-01-21T11:14:25.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:25.476 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:25.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc2bc02ff60 msgr2=0x7fc2bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc2bc02ff60 0x7fc2bc032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc2c80096a0 tx=0x7fc2c8006b40).stop 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 msgr2=0x7fc2d4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d4109430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc2c4013510 tx=0x7fc2c400b040).stop 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 shutdown_connections 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc2bc02ff60 0x7fc2bc032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 --2- 172.21.15.178:0/421385071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2d40f5410 0x7fc2d4109430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 >> 172.21.15.178:0/421385071 conn(0x7fc2d40f0b70 msgr2=0x7fc2d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 shutdown_connections 2024-01-21T11:14:25.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:25.476+0000 7fc2da47d700 1 -- 172.21.15.178:0/421385071 wait complete. 2024-01-21T11:14:25.479 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:25.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:25 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:25.476162+0000 mon.smithi089 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.178:0/421385071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:25.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:25 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:25.648608+0000 mgr.smithi089.ufybqw (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:26.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:26.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:27.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 -- 172.21.15.178:0/1355798072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 msgr2=0x7fe8f40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 --2- 172.21.15.178:0/1355798072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f40f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe8e4009fb0 tx=0x7fe8e4009b30).stop 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 -- 172.21.15.178:0/1355798072 shutdown_connections 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 --2- 172.21.15.178:0/1355798072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f40f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 -- 172.21.15.178:0/1355798072 >> 172.21.15.178:0/1355798072 conn(0x7fe8f40f09a0 msgr2=0x7fe8f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 -- 172.21.15.178:0/1355798072 shutdown_connections 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 -- 172.21.15.178:0/1355798072 wait complete. 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 Processor -- start 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.968+0000 7fe8fa29e700 1 -- start start 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8fa29e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8fa29e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4060240 con 0x7fe8f40f51a0 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8f929c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8f929c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56930/0 (socket says 172.21.15.178:56930) 2024-01-21T11:14:27.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8f929c700 1 -- 172.21.15.178:0/3606296204 learned_addr learned my addr 172.21.15.178:0/3606296204 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8f929c700 1 -- 172.21.15.178:0/3606296204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e4005040 con 0x7fe8f40f51a0 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8f929c700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f41091b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe8e4009830 tx=0x7fe8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e4009320 con 0x7fe8f40f51a0 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f4060440 con 0x7fe8f40f51a0 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8e40136c0 con 0x7fe8f40f51a0 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.969+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e401ee90 con 0x7fe8f40f51a0 2024-01-21T11:14:27.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.970+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f4109860 con 0x7fe8f40f51a0 2024-01-21T11:14:27.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.970+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8e401e420 con 0x7fe8f40f51a0 2024-01-21T11:14:27.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.970+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f4045ad0 con 0x7fe8f40f51a0 2024-01-21T11:14:27.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.970+0000 7fe8e9ffb700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8e002ff10 0x7fe8e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:27.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.970+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8e4041020 con 0x7fe8f40f51a0 2024-01-21T11:14:27.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.971+0000 7fe8f8a9b700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8e002ff10 0x7fe8e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:27.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.971+0000 7fe8f8a9b700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8e002ff10 0x7fe8e00323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe8f00096a0 tx=0x7fe8f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:27.976 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:27.975+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8e40095f0 con 0x7fe8f40f51a0 2024-01-21T11:14:28.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:27.648920+0000 mgr.smithi089.ufybqw (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:28.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.183+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8f410a070 con 0x7fe8f40f51a0 2024-01-21T11:14:28.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.184+0000 7fe8e9ffb700 1 -- 172.21.15.178:0/3606296204 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8e401c790 con 0x7fe8f40f51a0 2024-01-21T11:14:28.185 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:28.185 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:28.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8e002ff10 msgr2=0x7fe8e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:28.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8e002ff10 0x7fe8e00323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe8f00096a0 tx=0x7fe8f0006b40).stop 2024-01-21T11:14:28.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 msgr2=0x7fe8f41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:28.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f41091b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe8e4009830 tx=0x7fe8e400b040).stop 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 shutdown_connections 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8e002ff10 0x7fe8e00323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 --2- 172.21.15.178:0/3606296204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f40f51a0 0x7fe8f41091b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 >> 172.21.15.178:0/3606296204 conn(0x7fe8f40f09a0 msgr2=0x7fe8f40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 shutdown_connections 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:28.185+0000 7fe8fa29e700 1 -- 172.21.15.178:0/3606296204 wait complete. 2024-01-21T11:14:28.187 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:29.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:28.185400+0000 mon.smithi089 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.178:0/3606296204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:29.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:29.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 -- 172.21.15.178:0/3852654369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 msgr2=0x7fb4040f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 --2- 172.21.15.178:0/3852654369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f7d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb3ec009fb0 tx=0x7fb3ec009b30).stop 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 -- 172.21.15.178:0/3852654369 shutdown_connections 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 --2- 172.21.15.178:0/3852654369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f7d00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 -- 172.21.15.178:0/3852654369 >> 172.21.15.178:0/3852654369 conn(0x7fb4040f0120 msgr2=0x7fb4040f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 -- 172.21.15.178:0/3852654369 shutdown_connections 2024-01-21T11:14:30.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 -- 172.21.15.178:0/3852654369 wait complete. 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.741+0000 7fb408993700 1 Processor -- start 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb408993700 1 -- start start 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb408993700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb408993700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040f5630 con 0x7fb4040f78e0 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb402d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb402d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56952/0 (socket says 172.21.15.178:56952) 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb402d9d700 1 -- 172.21.15.178:0/1132865210 learned_addr learned my addr 172.21.15.178:0/1132865210 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.742+0000 7fb402d9d700 1 -- 172.21.15.178:0/1132865210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec005040 con 0x7fb4040f78e0 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb402d9d700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f71b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb3ec013510 tx=0x7fb3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:30.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ec009320 con 0x7fb4040f78e0 2024-01-21T11:14:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4040f57d0 con 0x7fb4040f78e0 2024-01-21T11:14:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3ec013f00 con 0x7fb4040f78e0 2024-01-21T11:14:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ec014dc0 con 0x7fb4040f78e0 2024-01-21T11:14:30.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4040f5c60 con 0x7fb4040f78e0 2024-01-21T11:14:30.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3ec0095f0 con 0x7fb4040f78e0 2024-01-21T11:14:30.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.743+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb404059100 con 0x7fb4040f78e0 2024-01-21T11:14:30.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.744+0000 7fb3fb7fe700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3f002ff60 0x7fb3f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:30.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.744+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3ec041040 con 0x7fb4040f78e0 2024-01-21T11:14:30.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.744+0000 7fb40259c700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3f002ff60 0x7fb3f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:30.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.744+0000 7fb40259c700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3f002ff60 0x7fb3f0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3f40096a0 tx=0x7fb3f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:30.749 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.748+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3ec014420 con 0x7fb4040f78e0 2024-01-21T11:14:30.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:29.649223+0000 mgr.smithi089.ufybqw (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:30.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.957+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4040fbce0 con 0x7fb4040f78e0 2024-01-21T11:14:30.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.958+0000 7fb3fb7fe700 1 -- 172.21.15.178:0/1132865210 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3ec010230 con 0x7fb4040f78e0 2024-01-21T11:14:30.959 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:30.959 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:30.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3f002ff60 msgr2=0x7fb3f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:30.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3f002ff60 0x7fb3f0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3f40096a0 tx=0x7fb3f4006b40).stop 2024-01-21T11:14:30.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 msgr2=0x7fb4040f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:30.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f71b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb3ec013510 tx=0x7fb3ec00b040).stop 2024-01-21T11:14:30.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 shutdown_connections 2024-01-21T11:14:30.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3f002ff60 0x7fb3f0032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:30.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 --2- 172.21.15.178:0/1132865210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4040f78e0 0x7fb4040f71b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:30.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 >> 172.21.15.178:0/1132865210 conn(0x7fb4040f0120 msgr2=0x7fb4040f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:30.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 shutdown_connections 2024-01-21T11:14:30.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:30.959+0000 7fb408993700 1 -- 172.21.15.178:0/1132865210 wait complete. 2024-01-21T11:14:30.961 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:31.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:31 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:30.959239+0000 mon.smithi089 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.178:0/1132865210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:32.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:32.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:32.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:31.649595+0000 mgr.smithi089.ufybqw (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:33.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 -- 172.21.15.178:0/712257994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 msgr2=0x7f51f80fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:33.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 --2- 172.21.15.178:0/712257994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f80fb550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f51e8009fb0 tx=0x7f51e8009b30).stop 2024-01-21T11:14:33.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 -- 172.21.15.178:0/712257994 shutdown_connections 2024-01-21T11:14:33.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 --2- 172.21.15.178:0/712257994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f80fb550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 -- 172.21.15.178:0/712257994 >> 172.21.15.178:0/712257994 conn(0x7f51f80f0b70 msgr2=0x7f51f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 -- 172.21.15.178:0/712257994 shutdown_connections 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.538+0000 7f51fe710700 1 -- 172.21.15.178:0/712257994 wait complete. 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51fe710700 1 Processor -- start 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51fe710700 1 -- start start 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51fe710700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f81088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51fe710700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8108db0 con 0x7f51f8060220 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51f7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f81088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51f7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f81088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58250/0 (socket says 172.21.15.178:58250) 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.539+0000 7f51f7fff700 1 -- 172.21.15.178:0/1851430088 learned_addr learned my addr 172.21.15.178:0/1851430088 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51f7fff700 1 -- 172.21.15.178:0/1851430088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e8005040 con 0x7f51f8060220 2024-01-21T11:14:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51f7fff700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f81088c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f51e8009830 tx=0x7f51e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:33.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e8009320 con 0x7f51f8060220 2024-01-21T11:14:33.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51e80136c0 con 0x7f51f8060220 2024-01-21T11:14:33.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f8108fb0 con 0x7f51f8060220 2024-01-21T11:14:33.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e801ee90 con 0x7f51f8060220 2024-01-21T11:14:33.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.540+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f8109440 con 0x7f51f8060220 2024-01-21T11:14:33.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.541+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51e801e420 con 0x7f51f8060220 2024-01-21T11:14:33.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.541+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f8045ad0 con 0x7f51f8060220 2024-01-21T11:14:33.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.541+0000 7f51f4ff9700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f51e002ff10 0x7f51e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:33.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.541+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51e8040fe0 con 0x7f51f8060220 2024-01-21T11:14:33.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.541+0000 7f51f77fe700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f51e002ff10 0x7f51e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:33.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.542+0000 7f51f77fe700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f51e002ff10 0x7f51e00323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f51ec0096a0 tx=0x7f51ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:33.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.545+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51e80095f0 con 0x7f51f8060220 2024-01-21T11:14:33.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.752+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51f810a2f0 con 0x7f51f8060220 2024-01-21T11:14:33.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.752+0000 7f51f4ff9700 1 -- 172.21.15.178:0/1851430088 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51e802c2d0 con 0x7f51f8060220 2024-01-21T11:14:33.754 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:33.754 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f51e002ff10 msgr2=0x7f51e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f51e002ff10 0x7f51e00323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f51ec0096a0 tx=0x7f51ec006b40).stop 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 msgr2=0x7f51f81088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f81088c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f51e8009830 tx=0x7f51e800b040).stop 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 shutdown_connections 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f51e002ff10 0x7f51e00323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 --2- 172.21.15.178:0/1851430088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f8060220 0x7f51f81088c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 >> 172.21.15.178:0/1851430088 conn(0x7f51f80f0b70 msgr2=0x7f51f80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 shutdown_connections 2024-01-21T11:14:33.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:33.754+0000 7f51fe710700 1 -- 172.21.15.178:0/1851430088 wait complete. 2024-01-21T11:14:33.756 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:34.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:33.649859+0000 mgr.smithi089.ufybqw (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:34.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:33.754267+0000 mon.smithi089 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.178:0/1851430088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:35.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:35.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:36.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.289+0000 7f9f94c63700 1 -- 172.21.15.178:0/2072286319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 msgr2=0x7f9f900f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:36.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.289+0000 7f9f94c63700 1 --2- 172.21.15.178:0/2072286319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f900f66c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f9f80009fb0 tx=0x7f9f80009b30).stop 2024-01-21T11:14:36.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 -- 172.21.15.178:0/2072286319 shutdown_connections 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 --2- 172.21.15.178:0/2072286319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f900f66c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 -- 172.21.15.178:0/2072286319 >> 172.21.15.178:0/2072286319 conn(0x7f9f900f1aa0 msgr2=0x7f9f900f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 -- 172.21.15.178:0/2072286319 shutdown_connections 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 -- 172.21.15.178:0/2072286319 wait complete. 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 Processor -- start 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 -- start start 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f901064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.290+0000 7f9f94c63700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f901069a0 con 0x7f9f900f62a0 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f8f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f901064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f8f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f901064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58274/0 (socket says 172.21.15.178:58274) 2024-01-21T11:14:36.292 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f8f7fe700 1 -- 172.21.15.178:0/2512297960 learned_addr learned my addr 172.21.15.178:0/2512297960 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f8f7fe700 1 -- 172.21.15.178:0/2512297960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80005040 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f8f7fe700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f901064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9f80013510 tx=0x7f9f8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f80009320 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f80013f00 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f90106ba0 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f80014dc0 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.291+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f90107030 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.292+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f80014420 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.292+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f90045ad0 con 0x7f9f900f62a0 2024-01-21T11:14:36.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.292+0000 7f9f77fff700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9f7802ff60 0x7f9f78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:36.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.292+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f800410f0 con 0x7f9f900f62a0 2024-01-21T11:14:36.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.292+0000 7f9f8effd700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9f7802ff60 0x7f9f78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:36.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.293+0000 7f9f8effd700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9f7802ff60 0x7f9f78032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9f840096a0 tx=0x7f9f84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:36.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.297+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f800095f0 con 0x7f9f900f62a0 2024-01-21T11:14:36.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.506+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f90107ee0 con 0x7f9f900f62a0 2024-01-21T11:14:36.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.507+0000 7f9f77fff700 1 -- 172.21.15.178:0/2512297960 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f8001cb10 con 0x7f9f900f62a0 2024-01-21T11:14:36.508 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:36.508 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:36.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9f7802ff60 msgr2=0x7f9f78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:36.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9f7802ff60 0x7f9f78032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9f840096a0 tx=0x7f9f84006b40).stop 2024-01-21T11:14:36.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 msgr2=0x7f9f901064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f901064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9f80013510 tx=0x7f9f8000b040).stop 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 shutdown_connections 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9f7802ff60 0x7f9f78032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 --2- 172.21.15.178:0/2512297960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f900f62a0 0x7f9f901064b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.508+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 >> 172.21.15.178:0/2512297960 conn(0x7f9f900f1aa0 msgr2=0x7f9f900f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.509+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 shutdown_connections 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:36.509+0000 7f9f94c63700 1 -- 172.21.15.178:0/2512297960 wait complete. 2024-01-21T11:14:36.510 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:36.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:35.650172+0000 mgr.smithi089.ufybqw (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:36.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:36.508413+0000 mon.smithi089 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.178:0/2512297960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:37.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:37.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:38.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:37.650419+0000 mgr.smithi089.ufybqw (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 -- 172.21.15.178:0/3157716718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 msgr2=0x7f63d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 --2- 172.21.15.178:0/3157716718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d00f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f63c0009fb0 tx=0x7f63c0009b30).stop 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 -- 172.21.15.178:0/3157716718 shutdown_connections 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 --2- 172.21.15.178:0/3157716718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d00f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 -- 172.21.15.178:0/3157716718 >> 172.21.15.178:0/3157716718 conn(0x7f63d00f0b70 msgr2=0x7f63d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 -- 172.21.15.178:0/3157716718 shutdown_connections 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.003+0000 7f63d79e8700 1 -- 172.21.15.178:0/3157716718 wait complete. 2024-01-21T11:14:39.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d79e8700 1 Processor -- start 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d79e8700 1 -- start start 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d79e8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d79e8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0060280 con 0x7f63d00f5410 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d5784700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d5784700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58296/0 (socket says 172.21.15.178:58296) 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.004+0000 7f63d5784700 1 -- 172.21.15.178:0/177407043 learned_addr learned my addr 172.21.15.178:0/177407043 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63d5784700 1 -- 172.21.15.178:0/177407043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0005040 con 0x7f63d00f5410 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63d5784700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f63c0013510 tx=0x7f63c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63c0009320 con 0x7f63d00f5410 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d0060480 con 0x7f63d00f5410 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63c0013f00 con 0x7f63d00f5410 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63c0014d90 con 0x7f63d00f5410 2024-01-21T11:14:39.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.005+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d0109b20 con 0x7f63d00f5410 2024-01-21T11:14:39.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.006+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63c00095f0 con 0x7f63d00f5410 2024-01-21T11:14:39.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.006+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d0045ad0 con 0x7f63d00f5410 2024-01-21T11:14:39.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.006+0000 7f63c5ffb700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63bc02ffb0 0x7f63bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:39.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.006+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63c00410c0 con 0x7f63d00f5410 2024-01-21T11:14:39.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.006+0000 7f63d4f83700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63bc02ffb0 0x7f63bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:39.009 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.007+0000 7f63d4f83700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63bc02ffb0 0x7f63bc032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f63cc0096a0 tx=0x7f63cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:39.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.011+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63c001d7f0 con 0x7f63d00f5410 2024-01-21T11:14:39.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.216+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63d010a340 con 0x7f63d00f5410 2024-01-21T11:14:39.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.217+0000 7f63c5ffb700 1 -- 172.21.15.178:0/177407043 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63c0010230 con 0x7f63d00f5410 2024-01-21T11:14:39.218 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:39.218 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:39.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63bc02ffb0 msgr2=0x7f63bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:39.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63bc02ffb0 0x7f63bc032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f63cc0096a0 tx=0x7f63cc006b40).stop 2024-01-21T11:14:39.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 msgr2=0x7f63d01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:39.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f63c0013510 tx=0x7f63c000b040).stop 2024-01-21T11:14:39.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 shutdown_connections 2024-01-21T11:14:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63bc02ffb0 0x7f63bc032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 --2- 172.21.15.178:0/177407043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 >> 172.21.15.178:0/177407043 conn(0x7f63d00f0b70 msgr2=0x7f63d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 shutdown_connections 2024-01-21T11:14:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:39.218+0000 7f63d79e8700 1 -- 172.21.15.178:0/177407043 wait complete. 2024-01-21T11:14:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:39.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:39.218343+0000 mon.smithi089 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.178:0/177407043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:40.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:40.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:40.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:39.650773+0000 mgr.smithi089.ufybqw (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/3732899844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 msgr2=0x7f7f480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 --2- 172.21.15.178:0/3732899844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f480f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7f38009fb0 tx=0x7f7f38009b30).stop 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/3732899844 shutdown_connections 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 --2- 172.21.15.178:0/3732899844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f480f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/3732899844 >> 172.21.15.178:0/3732899844 conn(0x7f7f480f0b70 msgr2=0x7f7f480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/3732899844 shutdown_connections 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.869+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/3732899844 wait complete. 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4fc6f700 1 Processor -- start 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4fc6f700 1 -- start start 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4fc6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f48109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4fc6f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48060240 con 0x7f7f480f5410 2024-01-21T11:14:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4da0b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f48109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4da0b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f48109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58320/0 (socket says 172.21.15.178:58320) 2024-01-21T11:14:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.870+0000 7f7f4da0b700 1 -- 172.21.15.178:0/2749091326 learned_addr learned my addr 172.21.15.178:0/2749091326 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f4da0b700 1 -- 172.21.15.178:0/2749091326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38005040 con 0x7f7f480f5410 2024-01-21T11:14:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f4da0b700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f48109430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7f38013510 tx=0x7f7f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f38009320 con 0x7f7f480f5410 2024-01-21T11:14:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f48060440 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f38013f00 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f38014d90 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.871+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f48109ae0 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.872+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f48045ad0 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.872+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f380095f0 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.872+0000 7f7f3e7fc700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7f3402ff60 0x7f7f34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.872+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f38040fc0 con 0x7f7f480f5410 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.872+0000 7f7f4d20a700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7f3402ff60 0x7f7f34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.873+0000 7f7f4d20a700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7f3402ff60 0x7f7f34032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7f440096a0 tx=0x7f7f44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:41.878+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f38014420 con 0x7f7f480f5410 2024-01-21T11:14:42.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.081+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f480fae60 con 0x7f7f480f5410 2024-01-21T11:14:42.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.081+0000 7f7f3e7fc700 1 -- 172.21.15.178:0/2749091326 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f38010230 con 0x7f7f480f5410 2024-01-21T11:14:42.082 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:42.082 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7f3402ff60 msgr2=0x7f7f34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7f3402ff60 0x7f7f34032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7f440096a0 tx=0x7f7f44006b40).stop 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 msgr2=0x7f7f48109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f48109430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7f38013510 tx=0x7f7f3800b040).stop 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 shutdown_connections 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7f3402ff60 0x7f7f34032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 --2- 172.21.15.178:0/2749091326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480f5410 0x7f7f48109430 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 >> 172.21.15.178:0/2749091326 conn(0x7f7f480f0b70 msgr2=0x7f7f480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 shutdown_connections 2024-01-21T11:14:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:42.083+0000 7f7f4fc6f700 1 -- 172.21.15.178:0/2749091326 wait complete. 2024-01-21T11:14:42.085 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:42.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:41.651088+0000 mgr.smithi089.ufybqw (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:42.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:42.083234+0000 mon.smithi089 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.178:0/2749091326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:43.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:43.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:44.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/2833261847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 msgr2=0x7f6df80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:44.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 --2- 172.21.15.178:0/2833261847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df80f90f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f6de8009fb0 tx=0x7f6de8009b30).stop 2024-01-21T11:14:44.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/2833261847 shutdown_connections 2024-01-21T11:14:44.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 --2- 172.21.15.178:0/2833261847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df80f90f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:44.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/2833261847 >> 172.21.15.178:0/2833261847 conn(0x7f6df80f0b70 msgr2=0x7f6df80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:44.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/2833261847 shutdown_connections 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/2833261847 wait complete. 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 Processor -- start 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- start start 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.638+0000 7f6dfcbf5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df8106bd0 con 0x7f6df8060280 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6df659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6df659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34210/0 (socket says 172.21.15.178:34210) 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6df659c700 1 -- 172.21.15.178:0/3169884904 learned_addr learned my addr 172.21.15.178:0/3169884904 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6df659c700 1 -- 172.21.15.178:0/3169884904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de8005040 con 0x7f6df8060280 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6df659c700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df81066e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f6de8013510 tx=0x7f6de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de8009320 con 0x7f6df8060280 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df8106dd0 con 0x7f6df8060280 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6de8013f00 con 0x7f6df8060280 2024-01-21T11:14:44.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de8014dc0 con 0x7f6df8060280 2024-01-21T11:14:44.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.639+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df8107260 con 0x7f6df8060280 2024-01-21T11:14:44.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.640+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df8045ad0 con 0x7f6df8060280 2024-01-21T11:14:44.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.640+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6de8014420 con 0x7f6df8060280 2024-01-21T11:14:44.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.640+0000 7f6de6ffd700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6de002ff60 0x7f6de0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:44.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.640+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6de80410a0 con 0x7f6df8060280 2024-01-21T11:14:44.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.640+0000 7f6df5d9b700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6de002ff60 0x7f6de0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:44.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.641+0000 7f6df5d9b700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6de002ff60 0x7f6de0032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6dec0096a0 tx=0x7f6dec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:44.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.645+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6de80095f0 con 0x7f6df8060280 2024-01-21T11:14:44.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.851+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6df8108110 con 0x7f6df8060280 2024-01-21T11:14:44.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.851+0000 7f6de6ffd700 1 -- 172.21.15.178:0/3169884904 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6de801cb10 con 0x7f6df8060280 2024-01-21T11:14:44.851 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:44.851 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6de002ff60 msgr2=0x7f6de0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6de002ff60 0x7f6de0032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6dec0096a0 tx=0x7f6dec006b40).stop 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 msgr2=0x7f6df81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df81066e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f6de8013510 tx=0x7f6de800b040).stop 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 shutdown_connections 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6de002ff60 0x7f6de0032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 --2- 172.21.15.178:0/3169884904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8060280 0x7f6df81066e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 >> 172.21.15.178:0/3169884904 conn(0x7f6df80f0b70 msgr2=0x7f6df80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 shutdown_connections 2024-01-21T11:14:44.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:44.853+0000 7f6dfcbf5700 1 -- 172.21.15.178:0/3169884904 wait complete. 2024-01-21T11:14:44.854 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:44.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:43.651437+0000 mgr.smithi089.ufybqw (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:45.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:44.851956+0000 mon.smithi089 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.178:0/3169884904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:46.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:46.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:46.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:45.651832+0000 mgr.smithi089.ufybqw (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- 172.21.15.178:0/1065267071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 msgr2=0x7f7a600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 --2- 172.21.15.178:0/1065267071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f7a5c009fb0 tx=0x7f7a5c009b30).stop 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- 172.21.15.178:0/1065267071 shutdown_connections 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 --2- 172.21.15.178:0/1065267071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- 172.21.15.178:0/1065267071 >> 172.21.15.178:0/1065267071 conn(0x7f7a600f09a0 msgr2=0x7f7a600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- 172.21.15.178:0/1065267071 shutdown_connections 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- 172.21.15.178:0/1065267071 wait complete. 2024-01-21T11:14:47.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 Processor -- start 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- start start 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.403+0000 7f7a66d58700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a600fb5a0 con 0x7f7a600f51a0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a65d56700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a65d56700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34220/0 (socket says 172.21.15.178:34220) 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a65d56700 1 -- 172.21.15.178:0/204130765 learned_addr learned my addr 172.21.15.178:0/204130765 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a65d56700 1 -- 172.21.15.178:0/204130765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c005040 con 0x7f7a600f51a0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a65d56700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600fd120 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f7a5c009830 tx=0x7f7a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c009320 con 0x7f7a600f51a0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a600fb740 con 0x7f7a600f51a0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a5c0136c0 con 0x7f7a600f51a0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c01ee90 con 0x7f7a600f51a0 2024-01-21T11:14:47.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.404+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a600fbbd0 con 0x7f7a600f51a0 2024-01-21T11:14:47.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.405+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a5c01e420 con 0x7f7a600f51a0 2024-01-21T11:14:47.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.405+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a60059100 con 0x7f7a600f51a0 2024-01-21T11:14:47.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.405+0000 7f7a567fc700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7a4c02ff10 0x7f7a4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:47.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.405+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a5c040f40 con 0x7f7a600f51a0 2024-01-21T11:14:47.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.405+0000 7f7a65555700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7a4c02ff10 0x7f7a4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:47.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.406+0000 7f7a65555700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7a4c02ff10 0x7f7a4c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7a500096a0 tx=0x7f7a50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:47.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.410+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a5c06d2b0 con 0x7f7a600f51a0 2024-01-21T11:14:47.620 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.620+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a60045ad0 con 0x7f7a600f51a0 2024-01-21T11:14:47.620 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.620+0000 7f7a567fc700 1 -- 172.21.15.178:0/204130765 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a5c0095f0 con 0x7f7a600f51a0 2024-01-21T11:14:47.620 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:47.621 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7a4c02ff10 msgr2=0x7f7a4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7a4c02ff10 0x7f7a4c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7a500096a0 tx=0x7f7a50006b40).stop 2024-01-21T11:14:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 msgr2=0x7f7a600fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600fd120 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f7a5c009830 tx=0x7f7a5c00b040).stop 2024-01-21T11:14:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 shutdown_connections 2024-01-21T11:14:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7a4c02ff10 0x7f7a4c0323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 --2- 172.21.15.178:0/204130765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a600f51a0 0x7f7a600fd120 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 >> 172.21.15.178:0/204130765 conn(0x7f7a600f09a0 msgr2=0x7f7a6005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 shutdown_connections 2024-01-21T11:14:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:47.622+0000 7f7a66d58700 1 -- 172.21.15.178:0/204130765 wait complete. 2024-01-21T11:14:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:47.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:47.621083+0000 mon.smithi089 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.178:0/204130765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:48.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:47.652137+0000 mgr.smithi089.ufybqw (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:48.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:48.998 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 -- 172.21.15.178:0/1827669643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 msgr2=0x7feee40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 --2- 172.21.15.178:0/1827669643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee40f90f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7feecc009fb0 tx=0x7feecc009b30).stop 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 -- 172.21.15.178:0/1827669643 shutdown_connections 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 --2- 172.21.15.178:0/1827669643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee40f90f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 -- 172.21.15.178:0/1827669643 >> 172.21.15.178:0/1827669643 conn(0x7feee40f0b70 msgr2=0x7feee40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 -- 172.21.15.178:0/1827669643 shutdown_connections 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.153+0000 7feee9fd9700 1 -- 172.21.15.178:0/1827669643 wait complete. 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee9fd9700 1 Processor -- start 2024-01-21T11:14:50.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee9fd9700 1 -- start start 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee9fd9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee9fd9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee4106bd0 con 0x7feee4060280 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34246/0 (socket says 172.21.15.178:34246) 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.154+0000 7feee37fe700 1 -- 172.21.15.178:0/4147788264 learned_addr learned my addr 172.21.15.178:0/4147788264 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feee37fe700 1 -- 172.21.15.178:0/4147788264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc005040 con 0x7feee4060280 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feee37fe700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee41066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7feecc013510 tx=0x7feecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feecc009320 con 0x7feee4060280 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee4106dd0 con 0x7feee4060280 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feecc013f00 con 0x7feee4060280 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feecc014dc0 con 0x7feee4060280 2024-01-21T11:14:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.155+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee4107260 con 0x7feee4060280 2024-01-21T11:14:50.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.156+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feecc014420 con 0x7feee4060280 2024-01-21T11:14:50.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.156+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee4045ad0 con 0x7feee4060280 2024-01-21T11:14:50.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.156+0000 7feedbfff700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7feed002ff60 0x7feed0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:50.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.156+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feecc041050 con 0x7feee4060280 2024-01-21T11:14:50.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.156+0000 7feee2ffd700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7feed002ff60 0x7feed0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:50.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.157+0000 7feee2ffd700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7feed002ff60 0x7feed0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feed40096a0 tx=0x7feed4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:50.161 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.161+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feecc0095f0 con 0x7feee4060280 2024-01-21T11:14:50.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.367+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feee4108110 con 0x7feee4060280 2024-01-21T11:14:50.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.368+0000 7feedbfff700 1 -- 172.21.15.178:0/4147788264 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feecc01cb10 con 0x7feee4060280 2024-01-21T11:14:50.367 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:50.368 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7feed002ff60 msgr2=0x7feed0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7feed002ff60 0x7feed0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feed40096a0 tx=0x7feed4006b40).stop 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 msgr2=0x7feee41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee41066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7feecc013510 tx=0x7feecc00b040).stop 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 shutdown_connections 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7feed002ff60 0x7feed0032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 --2- 172.21.15.178:0/4147788264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feee4060280 0x7feee41066e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 >> 172.21.15.178:0/4147788264 conn(0x7feee40f0b70 msgr2=0x7feee40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:50.369 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 shutdown_connections 2024-01-21T11:14:50.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:50.369+0000 7feee9fd9700 1 -- 172.21.15.178:0/4147788264 wait complete. 2024-01-21T11:14:50.370 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:50.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:49.652435+0000 mgr.smithi089.ufybqw (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:50.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:50.368302+0000 mon.smithi089 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.178:0/4147788264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:51.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:51.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.928+0000 7ff02bda5700 1 -- 172.21.15.178:0/1819087007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 msgr2=0x7ff0240f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.928+0000 7ff02bda5700 1 --2- 172.21.15.178:0/1819087007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff0240f5750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ff014009fb0 tx=0x7ff014009b30).stop 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 -- 172.21.15.178:0/1819087007 shutdown_connections 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 --2- 172.21.15.178:0/1819087007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff0240f5750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 -- 172.21.15.178:0/1819087007 >> 172.21.15.178:0/1819087007 conn(0x7ff0240f0a90 msgr2=0x7ff0240f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 -- 172.21.15.178:0/1819087007 shutdown_connections 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 -- 172.21.15.178:0/1819087007 wait complete. 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 Processor -- start 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 -- start start 2024-01-21T11:14:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff02410baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff02bda5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0241074e0 con 0x7ff0240f5330 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff029b41700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff02410baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff029b41700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff02410baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43214/0 (socket says 172.21.15.178:43214) 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.929+0000 7ff029b41700 1 -- 172.21.15.178:0/1580110759 learned_addr learned my addr 172.21.15.178:0/1580110759 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff029b41700 1 -- 172.21.15.178:0/1580110759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff014005040 con 0x7ff0240f5330 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff029b41700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff02410baf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff014014000 tx=0x7ff01400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0140095f0 con 0x7ff0240f5330 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff024060220 con 0x7ff0240f5330 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff014007e70 con 0x7ff0240f5330 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff014014ea0 con 0x7ff0240f5330 2024-01-21T11:14:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.930+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02410bfe0 con 0x7ff0240f5330 2024-01-21T11:14:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.931+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff014025070 con 0x7ff0240f5330 2024-01-21T11:14:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.931+0000 7ff01a7fc700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff01002ff60 0x7ff010032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.931+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024045ad0 con 0x7ff0240f5330 2024-01-21T11:14:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.931+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff014040a30 con 0x7ff0240f5330 2024-01-21T11:14:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.931+0000 7ff029340700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff01002ff60 0x7ff010032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.932+0000 7ff029340700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff01002ff60 0x7ff010032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff0200096a0 tx=0x7ff020006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:52.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:51.652747+0000 mgr.smithi089.ufybqw (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:52.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:52.935+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0140079c0 con 0x7ff0240f5330 2024-01-21T11:14:53.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.139+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff02405ea50 con 0x7ff0240f5330 2024-01-21T11:14:53.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.140+0000 7ff01a7fc700 1 -- 172.21.15.178:0/1580110759 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff01400f300 con 0x7ff0240f5330 2024-01-21T11:14:53.140 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:53.140 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff01002ff60 msgr2=0x7ff010032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff01002ff60 0x7ff010032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff0200096a0 tx=0x7ff020006b40).stop 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 msgr2=0x7ff02410baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff02410baf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff014014000 tx=0x7ff01400b040).stop 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 shutdown_connections 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff01002ff60 0x7ff010032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 --2- 172.21.15.178:0/1580110759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0240f5330 0x7ff02410baf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 >> 172.21.15.178:0/1580110759 conn(0x7ff0240f0a90 msgr2=0x7ff0240f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 shutdown_connections 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:53.142+0000 7ff02bda5700 1 -- 172.21.15.178:0/1580110759 wait complete. 2024-01-21T11:14:53.142 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:53.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:53.140715+0000 mon.smithi089 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.178:0/1580110759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:54.542 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:54.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:54.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:53.653028+0000 mgr.smithi089.ufybqw (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 -- 172.21.15.178:0/1258708505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 msgr2=0x7f23a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 --2- 172.21.15.178:0/1258708505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a80f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f239c009fb0 tx=0x7f239c009b30).stop 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 -- 172.21.15.178:0/1258708505 shutdown_connections 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 --2- 172.21.15.178:0/1258708505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a80f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 -- 172.21.15.178:0/1258708505 >> 172.21.15.178:0/1258708505 conn(0x7f23a80f1c70 msgr2=0x7f23a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 -- 172.21.15.178:0/1258708505 shutdown_connections 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.729+0000 7f23ad447700 1 -- 172.21.15.178:0/1258708505 wait complete. 2024-01-21T11:14:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23ad447700 1 Processor -- start 2024-01-21T11:14:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23ad447700 1 -- start start 2024-01-21T11:14:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23ad447700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23ad447700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8106c20 con 0x7f23a80f6510 2024-01-21T11:14:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23a6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23a6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43232/0 (socket says 172.21.15.178:43232) 2024-01-21T11:14:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.730+0000 7f23a6ffd700 1 -- 172.21.15.178:0/3537516325 learned_addr learned my addr 172.21.15.178:0/3537516325 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:55.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f23a6ffd700 1 -- 172.21.15.178:0/3537516325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c005040 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f23a6ffd700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a8106730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f239c013510 tx=0x7f239c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c009320 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a8106e20 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f239c013f00 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c014d90 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.731+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a81072b0 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.732+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f239c014420 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.732+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a8045ad0 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.732+0000 7f238f7fe700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f239002ffb0 0x7f2390032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.732+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f239c040fd0 con 0x7f23a80f6510 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.732+0000 7f23a67fc700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f239002ffb0 0x7f2390032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.733+0000 7f23a67fc700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f239002ffb0 0x7f2390032470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f23980096a0 tx=0x7f2398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:55.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.737+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f239c06d2b0 con 0x7f23a80f6510 2024-01-21T11:14:55.941 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.941+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23a80fbf60 con 0x7f23a80f6510 2024-01-21T11:14:55.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.942+0000 7f238f7fe700 1 -- 172.21.15.178:0/3537516325 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f239c026bf0 con 0x7f23a80f6510 2024-01-21T11:14:55.942 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:55.942 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:55.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.943+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f239002ffb0 msgr2=0x7f2390032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.943+0000 7f23ad447700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f239002ffb0 0x7f2390032470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f23980096a0 tx=0x7f2398006b40).stop 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.943+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 msgr2=0x7f23a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.943+0000 7f23ad447700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a8106730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f239c013510 tx=0x7f239c00b040).stop 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.944+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 shutdown_connections 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.944+0000 7f23ad447700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f239002ffb0 0x7f2390032470 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.944+0000 7f23ad447700 1 --2- 172.21.15.178:0/3537516325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a80f6510 0x7f23a8106730 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.944+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 >> 172.21.15.178:0/3537516325 conn(0x7f23a80f1c70 msgr2=0x7f23a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.944+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 shutdown_connections 2024-01-21T11:14:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:55.944+0000 7f23ad447700 1 -- 172.21.15.178:0/3537516325 wait complete. 2024-01-21T11:14:55.945 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:56.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:55.653358+0000 mgr.smithi089.ufybqw (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:57.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:55.942854+0000 mon.smithi089 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.178:0/3537516325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:14:57.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:14:57.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:14:58.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:57 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:57.653723+0000 mgr.smithi089.ufybqw (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:14:58.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 -- 172.21.15.178:0/421627835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 msgr2=0x7fba600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:58.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 --2- 172.21.15.178:0/421627835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba600f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fba54009fb0 tx=0x7fba54009b30).stop 2024-01-21T11:14:58.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 -- 172.21.15.178:0/421627835 shutdown_connections 2024-01-21T11:14:58.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 --2- 172.21.15.178:0/421627835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba600f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:58.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 -- 172.21.15.178:0/421627835 >> 172.21.15.178:0/421627835 conn(0x7fba600f1c70 msgr2=0x7fba600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:58.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 -- 172.21.15.178:0/421627835 shutdown_connections 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 -- 172.21.15.178:0/421627835 wait complete. 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 Processor -- start 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.562+0000 7fba67046700 1 -- start start 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba67046700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba6006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba67046700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba60069e20 con 0x7fba600f6510 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba64de2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba6006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba64de2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba6006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43254/0 (socket says 172.21.15.178:43254) 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba64de2700 1 -- 172.21.15.178:0/2815611788 learned_addr learned my addr 172.21.15.178:0/2815611788 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba64de2700 1 -- 172.21.15.178:0/2815611788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54005040 con 0x7fba600f6510 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba64de2700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba6006b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fba54013510 tx=0x7fba5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba54009320 con 0x7fba600f6510 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba60069fc0 con 0x7fba600f6510 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba54013f00 con 0x7fba600f6510 2024-01-21T11:14:58.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba54014d90 con 0x7fba600f6510 2024-01-21T11:14:58.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.563+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6006a450 con 0x7fba600f6510 2024-01-21T11:14:58.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.564+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba540263a0 con 0x7fba600f6510 2024-01-21T11:14:58.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.564+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba60059100 con 0x7fba600f6510 2024-01-21T11:14:58.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.564+0000 7fba5d7fa700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fba5002ff60 0x7fba50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:14:58.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.564+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba540412e0 con 0x7fba600f6510 2024-01-21T11:14:58.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.565+0000 7fba5ffff700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fba5002ff60 0x7fba50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:14:58.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.565+0000 7fba5ffff700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fba5002ff60 0x7fba50032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fba4c0096a0 tx=0x7fba4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:14:58.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.569+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba54012330 con 0x7fba600f6510 2024-01-21T11:14:58.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.772+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba600fbf60 con 0x7fba600f6510 2024-01-21T11:14:58.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.773+0000 7fba5d7fa700 1 -- 172.21.15.178:0/2815611788 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba54012330 con 0x7fba600f6510 2024-01-21T11:14:58.773 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:14:58.773 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:14:58.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fba5002ff60 msgr2=0x7fba50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:58.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fba5002ff60 0x7fba50032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fba4c0096a0 tx=0x7fba4c006b40).stop 2024-01-21T11:14:58.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 msgr2=0x7fba6006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:14:58.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba6006b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fba54013510 tx=0x7fba5400b040).stop 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 shutdown_connections 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fba5002ff60 0x7fba50032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 --2- 172.21.15.178:0/2815611788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba600f6510 0x7fba6006b9a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 >> 172.21.15.178:0/2815611788 conn(0x7fba600f1c70 msgr2=0x7fba600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 shutdown_connections 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:14:58.774+0000 7fba67046700 1 -- 172.21.15.178:0/2815611788 wait complete. 2024-01-21T11:14:58.775 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:14:59.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:14:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:14:58.773401+0000 mon.smithi089 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.178:0/2815611788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:00.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:00.180 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:00.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:14:59.654121+0000 mgr.smithi089.ufybqw (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 -- 172.21.15.178:0/1153400626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 msgr2=0x7f75f00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 --2- 172.21.15.178:0/1153400626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f00fb570 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f75e0009fb0 tx=0x7f75e0009b30).stop 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 -- 172.21.15.178:0/1153400626 shutdown_connections 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 --2- 172.21.15.178:0/1153400626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f00fb570 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 -- 172.21.15.178:0/1153400626 >> 172.21.15.178:0/1153400626 conn(0x7f75f00f0b70 msgr2=0x7f75f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 -- 172.21.15.178:0/1153400626 shutdown_connections 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.311+0000 7f75f7951700 1 -- 172.21.15.178:0/1153400626 wait complete. 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f7951700 1 Processor -- start 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f7951700 1 -- start start 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f7951700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f7951700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f0108dc0 con 0x7f75f0060200 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f56ed700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f56ed700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43274/0 (socket says 172.21.15.178:43274) 2024-01-21T11:15:01.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f56ed700 1 -- 172.21.15.178:0/252270003 learned_addr learned my addr 172.21.15.178:0/252270003 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.312+0000 7f75f56ed700 1 -- 172.21.15.178:0/252270003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e0005040 con 0x7f75f0060200 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.313+0000 7f75f56ed700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f01088d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f75e0013510 tx=0x7f75e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.313+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75e0009320 con 0x7f75f0060200 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.313+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f0108fc0 con 0x7f75f0060200 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.313+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75e0013f00 con 0x7f75f0060200 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.313+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75e0014dc0 con 0x7f75f0060200 2024-01-21T11:15:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.313+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f0109450 con 0x7f75f0060200 2024-01-21T11:15:01.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.314+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75e0014420 con 0x7f75f0060200 2024-01-21T11:15:01.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.314+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f0045ad0 con 0x7f75f0060200 2024-01-21T11:15:01.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.314+0000 7f75e5ffb700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f75dc02ff60 0x7f75dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:01.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.314+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75e00410e0 con 0x7f75f0060200 2024-01-21T11:15:01.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.314+0000 7f75f4eec700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f75dc02ff60 0x7f75dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:01.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.314+0000 7f75f4eec700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f75dc02ff60 0x7f75dc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f75ec0096a0 tx=0x7f75ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:01.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.318+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75e00095f0 con 0x7f75f0060200 2024-01-21T11:15:01.526 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.526+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75f010a300 con 0x7f75f0060200 2024-01-21T11:15:01.526 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.527+0000 7f75e5ffb700 1 -- 172.21.15.178:0/252270003 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75e001cb10 con 0x7f75f0060200 2024-01-21T11:15:01.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:01.527 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f75dc02ff60 msgr2=0x7f75dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f75dc02ff60 0x7f75dc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f75ec0096a0 tx=0x7f75ec006b40).stop 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 msgr2=0x7f75f01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f01088d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f75e0013510 tx=0x7f75e000b040).stop 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 shutdown_connections 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f75dc02ff60 0x7f75dc032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 --2- 172.21.15.178:0/252270003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75f0060200 0x7f75f01088d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:01.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 >> 172.21.15.178:0/252270003 conn(0x7f75f00f0b70 msgr2=0x7f75f00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:01.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 shutdown_connections 2024-01-21T11:15:01.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:01.528+0000 7f75f7951700 1 -- 172.21.15.178:0/252270003 wait complete. 2024-01-21T11:15:01.529 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:01.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:01.527289+0000 mon.smithi089 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.178:0/252270003' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:02.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:02.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:02.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:01.654475+0000 mgr.smithi089.ufybqw (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:04.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 -- 172.21.15.178:0/846810185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 msgr2=0x7f6c700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:04.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 --2- 172.21.15.178:0/846810185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c700f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f6c58009fb0 tx=0x7f6c58009b30).stop 2024-01-21T11:15:04.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 -- 172.21.15.178:0/846810185 shutdown_connections 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 --2- 172.21.15.178:0/846810185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c700f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 -- 172.21.15.178:0/846810185 >> 172.21.15.178:0/846810185 conn(0x7f6c700f1c70 msgr2=0x7f6c700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 -- 172.21.15.178:0/846810185 shutdown_connections 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.070+0000 7f6c7551d700 1 -- 172.21.15.178:0/846810185 wait complete. 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c7551d700 1 Processor -- start 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c7551d700 1 -- start start 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c7551d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c701066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c7551d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c70106bc0 con 0x7f6c700f6510 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c701066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c701066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49186/0 (socket says 172.21.15.178:49186) 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.071+0000 7f6c6effd700 1 -- 172.21.15.178:0/4121784283 learned_addr learned my addr 172.21.15.178:0/4121784283 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c6effd700 1 -- 172.21.15.178:0/4121784283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c58005040 con 0x7f6c700f6510 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c6effd700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c701066d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f6c58013510 tx=0x7f6c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c58009320 con 0x7f6c700f6510 2024-01-21T11:15:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c70106dc0 con 0x7f6c700f6510 2024-01-21T11:15:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c58013f00 con 0x7f6c700f6510 2024-01-21T11:15:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c58014d90 con 0x7f6c700f6510 2024-01-21T11:15:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.072+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c70107250 con 0x7f6c700f6510 2024-01-21T11:15:04.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.073+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c580263a0 con 0x7f6c700f6510 2024-01-21T11:15:04.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.073+0000 7f6c677fe700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c5c02ff60 0x7f6c5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:04.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.073+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c58041000 con 0x7f6c700f6510 2024-01-21T11:15:04.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.073+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c70045ad0 con 0x7f6c700f6510 2024-01-21T11:15:04.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.073+0000 7f6c6e7fc700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c5c02ff60 0x7f6c5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:04.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.074+0000 7f6c6e7fc700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c5c02ff60 0x7f6c5c032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f6c600096a0 tx=0x7f6c60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:04.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.078+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c58012920 con 0x7f6c700f6510 2024-01-21T11:15:04.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.284+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c700fbf60 con 0x7f6c700f6510 2024-01-21T11:15:04.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.285+0000 7f6c677fe700 1 -- 172.21.15.178:0/4121784283 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c5802b020 con 0x7f6c700f6510 2024-01-21T11:15:04.285 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:04.285 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:04.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c5c02ff60 msgr2=0x7f6c5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:04.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c5c02ff60 0x7f6c5c032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f6c600096a0 tx=0x7f6c60006b40).stop 2024-01-21T11:15:04.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 msgr2=0x7f6c701066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:04.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c701066d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f6c58013510 tx=0x7f6c5800b040).stop 2024-01-21T11:15:04.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 shutdown_connections 2024-01-21T11:15:04.287 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c5c02ff60 0x7f6c5c032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:04.287 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 --2- 172.21.15.178:0/4121784283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c700f6510 0x7f6c701066d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:04.287 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 >> 172.21.15.178:0/4121784283 conn(0x7f6c700f1c70 msgr2=0x7f6c700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:04.287 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 shutdown_connections 2024-01-21T11:15:04.287 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:04.286+0000 7f6c7551d700 1 -- 172.21.15.178:0/4121784283 wait complete. 2024-01-21T11:15:04.287 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:04.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:03.654851+0000 mgr.smithi089.ufybqw (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:04.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:04.285196+0000 mon.smithi089 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.178:0/4121784283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:05.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:05.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:06.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/160852393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 msgr2=0x7fd2bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:06.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 --2- 172.21.15.178:0/160852393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc0f55c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd2b8009fb0 tx=0x7fd2b8009b30).stop 2024-01-21T11:15:06.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/160852393 shutdown_connections 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 --2- 172.21.15.178:0/160852393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc0f55c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/160852393 >> 172.21.15.178:0/160852393 conn(0x7fd2bc0f09a0 msgr2=0x7fd2bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/160852393 shutdown_connections 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/160852393 wait complete. 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 Processor -- start 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- start start 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.884+0000 7fd2c2c2e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc060240 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c1c2c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c1c2c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49208/0 (socket says 172.21.15.178:49208) 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c1c2c700 1 -- 172.21.15.178:0/3817076 learned_addr learned my addr 172.21.15.178:0/3817076 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c1c2c700 1 -- 172.21.15.178:0/3817076 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8005040 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c1c2c700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc1091b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd2b8013510 tx=0x7fd2b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:06.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b8009320 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc060440 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2b8013f00 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b8014dc0 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.885+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc109860 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.886+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2b80095f0 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.886+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc045ad0 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.886+0000 7fd2b27fc700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd2a802ffb0 0x7fd2a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.886+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2b8040ff0 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.886+0000 7fd2c142b700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd2a802ffb0 0x7fd2a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.887+0000 7fd2c142b700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd2a802ffb0 0x7fd2a8032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fd2ac0096a0 tx=0x7fd2ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:06.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:06.891+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2b806d2b0 con 0x7fd2bc0f51a0 2024-01-21T11:15:06.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:05.655108+0000 mgr.smithi089.ufybqw (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:07.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.096+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2bc10a080 con 0x7fd2bc0f51a0 2024-01-21T11:15:07.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.097+0000 7fd2b27fc700 1 -- 172.21.15.178:0/3817076 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd2b8014420 con 0x7fd2bc0f51a0 2024-01-21T11:15:07.099 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:07.100 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:07.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd2a802ffb0 msgr2=0x7fd2a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd2a802ffb0 0x7fd2a8032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fd2ac0096a0 tx=0x7fd2ac006b40).stop 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 msgr2=0x7fd2bc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc1091b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd2b8013510 tx=0x7fd2b800b040).stop 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 shutdown_connections 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd2a802ffb0 0x7fd2a8032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 --2- 172.21.15.178:0/3817076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0f51a0 0x7fd2bc1091b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 >> 172.21.15.178:0/3817076 conn(0x7fd2bc0f09a0 msgr2=0x7fd2bc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 shutdown_connections 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:07.098+0000 7fd2c2c2e700 1 -- 172.21.15.178:0/3817076 wait complete. 2024-01-21T11:15:07.101 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:07.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:07 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:07.097596+0000 mon.smithi089 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.178:0/3817076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:08.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:08.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:08.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:07.655435+0000 mgr.smithi089.ufybqw (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:09.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 -- 172.21.15.178:0/1998266801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 msgr2=0x7f06340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:09.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 --2- 172.21.15.178:0/1998266801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f06340f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0624009fb0 tx=0x7f0624009b30).stop 2024-01-21T11:15:09.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 -- 172.21.15.178:0/1998266801 shutdown_connections 2024-01-21T11:15:09.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 --2- 172.21.15.178:0/1998266801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f06340f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:09.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 -- 172.21.15.178:0/1998266801 >> 172.21.15.178:0/1998266801 conn(0x7f06340f0b70 msgr2=0x7f06340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 -- 172.21.15.178:0/1998266801 shutdown_connections 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.632+0000 7f063bea2700 1 -- 172.21.15.178:0/1998266801 wait complete. 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f063bea2700 1 Processor -- start 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f063bea2700 1 -- start start 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f063bea2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f0634106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f063bea2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634106c10 con 0x7f06340f5410 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f0639c3e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f0634106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f0639c3e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f0634106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49220/0 (socket says 172.21.15.178:49220) 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.633+0000 7f0639c3e700 1 -- 172.21.15.178:0/3399297391 learned_addr learned my addr 172.21.15.178:0/3399297391 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f0639c3e700 1 -- 172.21.15.178:0/3399297391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0624005040 con 0x7f06340f5410 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f0639c3e700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f0634106720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0624013510 tx=0x7f062400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0624009320 con 0x7f06340f5410 2024-01-21T11:15:09.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0634106e10 con 0x7f06340f5410 2024-01-21T11:15:09.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0624013f00 con 0x7f06340f5410 2024-01-21T11:15:09.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0624014d90 con 0x7f06340f5410 2024-01-21T11:15:09.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.634+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06341072a0 con 0x7f06340f5410 2024-01-21T11:15:09.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.635+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06240095f0 con 0x7f06340f5410 2024-01-21T11:15:09.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.635+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634045ad0 con 0x7f06340f5410 2024-01-21T11:15:09.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.635+0000 7f062a7fc700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f062002ff60 0x7f0620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:09.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.635+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0624041000 con 0x7f06340f5410 2024-01-21T11:15:09.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.635+0000 7f063943d700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f062002ff60 0x7f0620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:09.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.636+0000 7f063943d700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f062002ff60 0x7f0620032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f06300096a0 tx=0x7f0630006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:09.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.640+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f062401d7f0 con 0x7f06340f5410 2024-01-21T11:15:09.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.844+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0634108150 con 0x7f06340f5410 2024-01-21T11:15:09.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.845+0000 7f062a7fc700 1 -- 172.21.15.178:0/3399297391 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0624010230 con 0x7f06340f5410 2024-01-21T11:15:09.845 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:09.845 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:09.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.846+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f062002ff60 msgr2=0x7f0620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.846+0000 7f063bea2700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f062002ff60 0x7f0620032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f06300096a0 tx=0x7f0630006b40).stop 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.846+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 msgr2=0x7f0634106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.846+0000 7f063bea2700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f0634106720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0624013510 tx=0x7f062400b040).stop 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.847+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 shutdown_connections 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.847+0000 7f063bea2700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f062002ff60 0x7f0620032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.847+0000 7f063bea2700 1 --2- 172.21.15.178:0/3399297391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06340f5410 0x7f0634106720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.847+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 >> 172.21.15.178:0/3399297391 conn(0x7f06340f0b70 msgr2=0x7f063405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.847+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 shutdown_connections 2024-01-21T11:15:09.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:09.847+0000 7f063bea2700 1 -- 172.21.15.178:0/3399297391 wait complete. 2024-01-21T11:15:09.848 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:10.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:09.655641+0000 mgr.smithi089.ufybqw (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:10.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:09.845750+0000 mon.smithi089 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.178:0/3399297391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:11.248 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:11.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:12.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 -- 172.21.15.178:0/4238705710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 msgr2=0x7fa1140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 --2- 172.21.15.178:0/4238705710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa108009fb0 tx=0x7fa108009b30).stop 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 -- 172.21.15.178:0/4238705710 shutdown_connections 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 --2- 172.21.15.178:0/4238705710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 -- 172.21.15.178:0/4238705710 >> 172.21.15.178:0/4238705710 conn(0x7fa1140f0b70 msgr2=0x7fa1140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 -- 172.21.15.178:0/4238705710 shutdown_connections 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 -- 172.21.15.178:0/4238705710 wait complete. 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.357+0000 7fa119d6f700 1 Processor -- start 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa119d6f700 1 -- start start 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa119d6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:12.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa119d6f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140fb820 con 0x7fa1140f5410 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa1137fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa1137fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:50400/0 (socket says 172.21.15.178:50400) 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa1137fe700 1 -- 172.21.15.178:0/3590712458 learned_addr learned my addr 172.21.15.178:0/3590712458 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.358+0000 7fa1137fe700 1 -- 172.21.15.178:0/3590712458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108005040 con 0x7fa1140f5410 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.359+0000 7fa1137fe700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140fd3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa108013510 tx=0x7fa10800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.359+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa108009320 con 0x7fa1140f5410 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.359+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140fb9c0 con 0x7fa1140f5410 2024-01-21T11:15:12.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.359+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa108013f00 con 0x7fa1140f5410 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.359+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa108014d90 con 0x7fa1140f5410 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.359+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1140fbe50 con 0x7fa1140f5410 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.360+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1080095f0 con 0x7fa1140f5410 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.360+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa114059100 con 0x7fa1140f5410 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.360+0000 7fa0fbfff700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa0fc02ff60 0x7fa0fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.360+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa108041030 con 0x7fa1140f5410 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.360+0000 7fa112ffd700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa0fc02ff60 0x7fa0fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:12.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.361+0000 7fa112ffd700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa0fc02ff60 0x7fa0fc032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa1040096a0 tx=0x7fa104006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:12.364 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.365+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa108014420 con 0x7fa1140f5410 2024-01-21T11:15:12.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.574+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1140fae60 con 0x7fa1140f5410 2024-01-21T11:15:12.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.575+0000 7fa0fbfff700 1 -- 172.21.15.178:0/3590712458 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa108010230 con 0x7fa1140f5410 2024-01-21T11:15:12.575 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:12.575 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:12.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.576+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa0fc02ff60 msgr2=0x7fa0fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.576+0000 7fa119d6f700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa0fc02ff60 0x7fa0fc032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa1040096a0 tx=0x7fa104006b40).stop 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.576+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 msgr2=0x7fa1140fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.576+0000 7fa119d6f700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140fd3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa108013510 tx=0x7fa10800b040).stop 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.577+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 shutdown_connections 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.577+0000 7fa119d6f700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa0fc02ff60 0x7fa0fc032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.577+0000 7fa119d6f700 1 --2- 172.21.15.178:0/3590712458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1140f5410 0x7fa1140fd3a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.577+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 >> 172.21.15.178:0/3590712458 conn(0x7fa1140f0b70 msgr2=0x7fa11405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.577+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 shutdown_connections 2024-01-21T11:15:12.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:12.577+0000 7fa119d6f700 1 -- 172.21.15.178:0/3590712458 wait complete. 2024-01-21T11:15:12.578 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:12.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:11.655972+0000 mgr.smithi089.ufybqw (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:12.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:12.575593+0000 mon.smithi089 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.178:0/3590712458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:13.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:13.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:13.656264+0000 mgr.smithi089.ufybqw (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:13.772376+0000 mon.smithi089 (mon.0) 150 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:15:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:13.781684+0000 mon.smithi089 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3264914132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 msgr2=0x7fa4b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 --2- 172.21.15.178:0/3264914132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b00f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa4a4009fb0 tx=0x7fa4a4009b30).stop 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3264914132 shutdown_connections 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 --2- 172.21.15.178:0/3264914132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b00f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3264914132 >> 172.21.15.178:0/3264914132 conn(0x7fa4b00f0b70 msgr2=0x7fa4b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3264914132 shutdown_connections 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.072+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3264914132 wait complete. 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4b57a9700 1 Processor -- start 2024-01-21T11:15:15.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4b57a9700 1 -- start start 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4b57a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4b57a9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0105af0 con 0x7fa4b00f5410 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:50430/0 (socket says 172.21.15.178:50430) 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.073+0000 7fa4aeffd700 1 -- 172.21.15.178:0/3444852974 learned_addr learned my addr 172.21.15.178:0/3444852974 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4aeffd700 1 -- 172.21.15.178:0/3444852974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4005040 con 0x7fa4b00f5410 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4aeffd700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b0105600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fa4a4013510 tx=0x7fa4a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a4009320 con 0x7fa4b00f5410 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4a4013f00 con 0x7fa4b00f5410 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4a4014d90 con 0x7fa4b00f5410 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b0105cf0 con 0x7fa4b00f5410 2024-01-21T11:15:15.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.074+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b0106180 con 0x7fa4b00f5410 2024-01-21T11:15:15.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.075+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4a40095f0 con 0x7fa4b00f5410 2024-01-21T11:15:15.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.075+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b0045ad0 con 0x7fa4b00f5410 2024-01-21T11:15:15.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.075+0000 7fa4977fe700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa49802ff60 0x7fa498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:15.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.075+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4a4041000 con 0x7fa4b00f5410 2024-01-21T11:15:15.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.075+0000 7fa4ae7fc700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa49802ff60 0x7fa498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:15.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.076+0000 7fa4ae7fc700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa49802ff60 0x7fa498032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa4a00096a0 tx=0x7fa4a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:15.080 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.080+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4a401d7f0 con 0x7fa4b00f5410 2024-01-21T11:15:15.288 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.287+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4b00fae60 con 0x7fa4b00f5410 2024-01-21T11:15:15.289 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.288+0000 7fa4977fe700 1 -- 172.21.15.178:0/3444852974 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4a4010230 con 0x7fa4b00f5410 2024-01-21T11:15:15.289 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:15.289 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:15.290 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.289+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa49802ff60 msgr2=0x7fa498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:15.290 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.289+0000 7fa4b57a9700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa49802ff60 0x7fa498032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa4a00096a0 tx=0x7fa4a0006b40).stop 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.289+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 msgr2=0x7fa4b0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.289+0000 7fa4b57a9700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b0105600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fa4a4013510 tx=0x7fa4a400b040).stop 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.290+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 shutdown_connections 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.290+0000 7fa4b57a9700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa49802ff60 0x7fa498032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.290+0000 7fa4b57a9700 1 --2- 172.21.15.178:0/3444852974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4b00f5410 0x7fa4b0105600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.290+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 >> 172.21.15.178:0/3444852974 conn(0x7fa4b00f0b70 msgr2=0x7fa4b00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.290+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 shutdown_connections 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:15.290+0000 7fa4b57a9700 1 -- 172.21.15.178:0/3444852974 wait complete. 2024-01-21T11:15:15.291 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:15.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:15.289747+0000 mon.smithi089 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.178:0/3444852974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:16.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:16.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:16.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:15.656657+0000 mgr.smithi089.ufybqw (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:17.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.891+0000 7f65892c6700 1 -- 172.21.15.178:0/3450391016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 msgr2=0x7f65840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:17.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.891+0000 7f65892c6700 1 --2- 172.21.15.178:0/3450391016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65840f90f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6574009fb0 tx=0x7f6574009b30).stop 2024-01-21T11:15:17.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 -- 172.21.15.178:0/3450391016 shutdown_connections 2024-01-21T11:15:17.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 --2- 172.21.15.178:0/3450391016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65840f90f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:17.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 -- 172.21.15.178:0/3450391016 >> 172.21.15.178:0/3450391016 conn(0x7f65840f0b70 msgr2=0x7f65840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 -- 172.21.15.178:0/3450391016 shutdown_connections 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 -- 172.21.15.178:0/3450391016 wait complete. 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 Processor -- start 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 -- start start 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.892+0000 7f65892c6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584106bd0 con 0x7f6584060280 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f6582ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f6582ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:50440/0 (socket says 172.21.15.178:50440) 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f6582ffd700 1 -- 172.21.15.178:0/20289898 learned_addr learned my addr 172.21.15.178:0/20289898 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f6582ffd700 1 -- 172.21.15.178:0/20289898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6574005040 con 0x7f6584060280 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f6582ffd700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65841066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6574013510 tx=0x7f657400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6574009320 con 0x7f6584060280 2024-01-21T11:15:17.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6584106dd0 con 0x7f6584060280 2024-01-21T11:15:17.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6574013f00 con 0x7f6584060280 2024-01-21T11:15:17.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6574014dc0 con 0x7f6584060280 2024-01-21T11:15:17.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.893+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6584107260 con 0x7f6584060280 2024-01-21T11:15:17.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.894+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6574014420 con 0x7f6584060280 2024-01-21T11:15:17.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.894+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6584045ad0 con 0x7f6584060280 2024-01-21T11:15:17.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.894+0000 7f657b7fe700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f657002ff60 0x7f6570032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:17.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.894+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65740410d0 con 0x7f6584060280 2024-01-21T11:15:17.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.895+0000 7f657bfff700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f657002ff60 0x7f6570032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:17.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.895+0000 7f657bfff700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f657002ff60 0x7f6570032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f656c0096a0 tx=0x7f656c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:17.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:17.899+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65740095f0 con 0x7f6584060280 2024-01-21T11:15:18.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.103+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6584108110 con 0x7f6584060280 2024-01-21T11:15:18.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.104+0000 7f657b7fe700 1 -- 172.21.15.178:0/20289898 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f657401cb10 con 0x7f6584060280 2024-01-21T11:15:18.105 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:18.105 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f657002ff60 msgr2=0x7f6570032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f657002ff60 0x7f6570032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f656c0096a0 tx=0x7f656c006b40).stop 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 msgr2=0x7f65841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65841066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6574013510 tx=0x7f657400b040).stop 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 shutdown_connections 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f657002ff60 0x7f6570032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 --2- 172.21.15.178:0/20289898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6584060280 0x7f65841066e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:18.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 >> 172.21.15.178:0/20289898 conn(0x7f65840f0b70 msgr2=0x7f65840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:18.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 shutdown_connections 2024-01-21T11:15:18.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:18.106+0000 7f65892c6700 1 -- 172.21.15.178:0/20289898 wait complete. 2024-01-21T11:15:18.108 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:18.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:17.656994+0000 mgr.smithi089.ufybqw (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:18.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:18.105767+0000 mon.smithi089 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.178:0/20289898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:19.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:19.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:20.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.724+0000 7f63fca50700 1 -- 172.21.15.178:0/1371035777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 msgr2=0x7f63f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:20.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.724+0000 7f63fca50700 1 --2- 172.21.15.178:0/1371035777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f80f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f63ec009fb0 tx=0x7f63ec009b30).stop 2024-01-21T11:15:20.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 -- 172.21.15.178:0/1371035777 shutdown_connections 2024-01-21T11:15:20.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 --2- 172.21.15.178:0/1371035777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f80f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 -- 172.21.15.178:0/1371035777 >> 172.21.15.178:0/1371035777 conn(0x7f63f80f0b70 msgr2=0x7f63f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 -- 172.21.15.178:0/1371035777 shutdown_connections 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 -- 172.21.15.178:0/1371035777 wait complete. 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 Processor -- start 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 -- start start 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.725+0000 7f63fca50700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8106c10 con 0x7f63f80f5410 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63f659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63f659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:50460/0 (socket says 172.21.15.178:50460) 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63f659c700 1 -- 172.21.15.178:0/3670788693 learned_addr learned my addr 172.21.15.178:0/3670788693 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63f659c700 1 -- 172.21.15.178:0/3670788693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec005040 con 0x7f63f80f5410 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63f659c700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f8106720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f63ec013510 tx=0x7f63ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:20.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63ec009320 con 0x7f63f80f5410 2024-01-21T11:15:20.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f8106e10 con 0x7f63f80f5410 2024-01-21T11:15:20.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63ec013f00 con 0x7f63f80f5410 2024-01-21T11:15:20.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63ec014d90 con 0x7f63f80f5410 2024-01-21T11:15:20.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.726+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f81072a0 con 0x7f63f80f5410 2024-01-21T11:15:20.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.727+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63ec014420 con 0x7f63f80f5410 2024-01-21T11:15:20.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.727+0000 7f63e6ffd700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63e002ff60 0x7f63e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:20.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.727+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8045ad0 con 0x7f63f80f5410 2024-01-21T11:15:20.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.727+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63ec041070 con 0x7f63f80f5410 2024-01-21T11:15:20.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.728+0000 7f63f5d9b700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63e002ff60 0x7f63e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:20.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.728+0000 7f63f5d9b700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63e002ff60 0x7f63e0032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f63e80096a0 tx=0x7f63e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:20.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.732+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63ec0095f0 con 0x7f63f80f5410 2024-01-21T11:15:20.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:19.657364+0000 mgr.smithi089.ufybqw (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:20.940 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.939+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63f8108150 con 0x7f63f80f5410 2024-01-21T11:15:20.941 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.940+0000 7f63e6ffd700 1 -- 172.21.15.178:0/3670788693 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63ec01db70 con 0x7f63f80f5410 2024-01-21T11:15:20.941 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:20.941 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63e002ff60 msgr2=0x7f63e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63e002ff60 0x7f63e0032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f63e80096a0 tx=0x7f63e8006b40).stop 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 msgr2=0x7f63f8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f8106720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f63ec013510 tx=0x7f63ec00b040).stop 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 shutdown_connections 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63e002ff60 0x7f63e0032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 --2- 172.21.15.178:0/3670788693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80f5410 0x7f63f8106720 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 >> 172.21.15.178:0/3670788693 conn(0x7f63f80f0b70 msgr2=0x7f63f805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:20.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 shutdown_connections 2024-01-21T11:15:20.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:20.942+0000 7f63fca50700 1 -- 172.21.15.178:0/3670788693 wait complete. 2024-01-21T11:15:20.944 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:22.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:20.941708+0000 mon.smithi089 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.178:0/3670788693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:22.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:22.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:21.657714+0000 mgr.smithi089.ufybqw (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 -- 172.21.15.178:0/634091345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 msgr2=0x7f40a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 --2- 172.21.15.178:0/634091345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a40f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f4094009fb0 tx=0x7f4094009b30).stop 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 -- 172.21.15.178:0/634091345 shutdown_connections 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 --2- 172.21.15.178:0/634091345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a40f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 -- 172.21.15.178:0/634091345 >> 172.21.15.178:0/634091345 conn(0x7f40a40f09a0 msgr2=0x7f40a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 -- 172.21.15.178:0/634091345 shutdown_connections 2024-01-21T11:15:23.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.549+0000 7f40a3fff700 1 -- 172.21.15.178:0/634091345 wait complete. 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a3fff700 1 Processor -- start 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a3fff700 1 -- start start 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a3fff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4060220 con 0x7f40a40f51a0 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58172/0 (socket says 172.21.15.178:58172) 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.550+0000 7f40a2ffd700 1 -- 172.21.15.178:0/3616094839 learned_addr learned my addr 172.21.15.178:0/3616094839 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f40a2ffd700 1 -- 172.21.15.178:0/3616094839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094005040 con 0x7f40a40f51a0 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f40a2ffd700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a41091d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4094013510 tx=0x7f409400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4094009320 con 0x7f40a40f51a0 2024-01-21T11:15:23.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a4060420 con 0x7f40a40f51a0 2024-01-21T11:15:23.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4094013f00 con 0x7f40a40f51a0 2024-01-21T11:15:23.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4094014dc0 con 0x7f40a40f51a0 2024-01-21T11:15:23.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.551+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a4109880 con 0x7f40a40f51a0 2024-01-21T11:15:23.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.552+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40940095f0 con 0x7f40a40f51a0 2024-01-21T11:15:23.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.552+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a4059100 con 0x7f40a40f51a0 2024-01-21T11:15:23.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.552+0000 7f408b7fe700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f408c02ff60 0x7f408c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:23.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.552+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4094041000 con 0x7f40a40f51a0 2024-01-21T11:15:23.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.552+0000 7f40a27fc700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f408c02ff60 0x7f408c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:23.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.553+0000 7f40a27fc700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f408c02ff60 0x7f408c032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f40980096a0 tx=0x7f4098006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:23.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.557+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4094014420 con 0x7f40a40f51a0 2024-01-21T11:15:23.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.762+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40a4045ad0 con 0x7f40a40f51a0 2024-01-21T11:15:23.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.763+0000 7f408b7fe700 1 -- 172.21.15.178:0/3616094839 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4094010230 con 0x7f40a40f51a0 2024-01-21T11:15:23.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:23.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f408c02ff60 msgr2=0x7f408c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:23.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f408c02ff60 0x7f408c032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f40980096a0 tx=0x7f4098006b40).stop 2024-01-21T11:15:23.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 msgr2=0x7f40a41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:23.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a41091d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4094013510 tx=0x7f409400b040).stop 2024-01-21T11:15:23.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 shutdown_connections 2024-01-21T11:15:23.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f408c02ff60 0x7f408c032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:23.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 --2- 172.21.15.178:0/3616094839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a40f51a0 0x7f40a41091d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:23.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 >> 172.21.15.178:0/3616094839 conn(0x7f40a40f09a0 msgr2=0x7f40a40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:23.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 shutdown_connections 2024-01-21T11:15:23.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:23.764+0000 7f40a3fff700 1 -- 172.21.15.178:0/3616094839 wait complete. 2024-01-21T11:15:23.766 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:25.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:23.658071+0000 mgr.smithi089.ufybqw (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:25.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:23.764644+0000 mon.smithi089 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.178:0/3616094839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:25.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:25.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:26.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:25 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:25.658414+0000 mgr.smithi089.ufybqw (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 -- 172.21.15.178:0/1673858960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 msgr2=0x7fd2380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 --2- 172.21.15.178:0/1673858960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd2380f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd22c009fb0 tx=0x7fd22c009b30).stop 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 -- 172.21.15.178:0/1673858960 shutdown_connections 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 --2- 172.21.15.178:0/1673858960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd2380f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 -- 172.21.15.178:0/1673858960 >> 172.21.15.178:0/1673858960 conn(0x7fd2380f1c70 msgr2=0x7fd2380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 -- 172.21.15.178:0/1673858960 shutdown_connections 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.336+0000 7fd23d88f700 1 -- 172.21.15.178:0/1673858960 wait complete. 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd23d88f700 1 Processor -- start 2024-01-21T11:15:26.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd23d88f700 1 -- start start 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd23d88f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd23806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd23d88f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd238069e20 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd236ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd23806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd236ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd23806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58192/0 (socket says 172.21.15.178:58192) 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.337+0000 7fd236ffd700 1 -- 172.21.15.178:0/1101717682 learned_addr learned my addr 172.21.15.178:0/1101717682 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd236ffd700 1 -- 172.21.15.178:0/1101717682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c005040 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd236ffd700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd23806b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd22c013510 tx=0x7fd22c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd22c009320 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd22c013f00 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd22c014d90 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd238069fc0 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.338+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23806a450 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.339+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd22c0095f0 con 0x7fd2380f6510 2024-01-21T11:15:26.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.339+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238059100 con 0x7fd2380f6510 2024-01-21T11:15:26.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.339+0000 7fd21f7fe700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd22002ff60 0x7fd220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:26.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.339+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd22c041020 con 0x7fd2380f6510 2024-01-21T11:15:26.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.339+0000 7fd2367fc700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd22002ff60 0x7fd220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:26.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.340+0000 7fd2367fc700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd22002ff60 0x7fd220032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fd2280096a0 tx=0x7fd228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:26.344 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.344+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd22c014420 con 0x7fd2380f6510 2024-01-21T11:15:26.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.548+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2380fbf60 con 0x7fd2380f6510 2024-01-21T11:15:26.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.549+0000 7fd21f7fe700 1 -- 172.21.15.178:0/1101717682 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd22c01d7f0 con 0x7fd2380f6510 2024-01-21T11:15:26.550 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:26.550 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:26.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd22002ff60 msgr2=0x7fd220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd22002ff60 0x7fd220032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fd2280096a0 tx=0x7fd228006b40).stop 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 msgr2=0x7fd23806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd23806b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd22c013510 tx=0x7fd22c00b040).stop 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 shutdown_connections 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd22002ff60 0x7fd220032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 --2- 172.21.15.178:0/1101717682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2380f6510 0x7fd23806b9a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.550+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 >> 172.21.15.178:0/1101717682 conn(0x7fd2380f1c70 msgr2=0x7fd2380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.551+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 shutdown_connections 2024-01-21T11:15:26.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:26.551+0000 7fd23d88f700 1 -- 172.21.15.178:0/1101717682 wait complete. 2024-01-21T11:15:26.553 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:27.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:26.550500+0000 mon.smithi089 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.178:0/1101717682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:27.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:27.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:28.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:27.658692+0000 mgr.smithi089.ufybqw (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 -- 172.21.15.178:0/2267866502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 msgr2=0x7f48880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 --2- 172.21.15.178:0/2267866502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48880f55c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4878009fb0 tx=0x7f4878009b30).stop 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 -- 172.21.15.178:0/2267866502 shutdown_connections 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 --2- 172.21.15.178:0/2267866502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48880f55c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 -- 172.21.15.178:0/2267866502 >> 172.21.15.178:0/2267866502 conn(0x7f48880f09a0 msgr2=0x7f48880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 -- 172.21.15.178:0/2267866502 shutdown_connections 2024-01-21T11:15:29.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.105+0000 7f4887fff700 1 -- 172.21.15.178:0/2267866502 wait complete. 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4887fff700 1 Processor -- start 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4887fff700 1 -- start start 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4887fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48881091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4887fff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888060240 con 0x7f48880f51a0 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4886ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48881091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4886ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48881091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58204/0 (socket says 172.21.15.178:58204) 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.106+0000 7f4886ffd700 1 -- 172.21.15.178:0/2969926401 learned_addr learned my addr 172.21.15.178:0/2969926401 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f4886ffd700 1 -- 172.21.15.178:0/2969926401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878005040 con 0x7f48880f51a0 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f4886ffd700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48881091e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4878013510 tx=0x7f487800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:29.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4878009320 con 0x7f48880f51a0 2024-01-21T11:15:29.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4888060440 con 0x7f48880f51a0 2024-01-21T11:15:29.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4878013f00 con 0x7f48880f51a0 2024-01-21T11:15:29.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4878014dc0 con 0x7f48880f51a0 2024-01-21T11:15:29.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.107+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4888109890 con 0x7f48880f51a0 2024-01-21T11:15:29.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.108+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4878014420 con 0x7f48880f51a0 2024-01-21T11:15:29.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.108+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4888045ad0 con 0x7f48880f51a0 2024-01-21T11:15:29.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.108+0000 7f486f7fe700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f487002ff60 0x7f4870032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:29.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.108+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48780410d0 con 0x7f48880f51a0 2024-01-21T11:15:29.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.108+0000 7f48867fc700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f487002ff60 0x7f4870032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:29.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.109+0000 7f48867fc700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f487002ff60 0x7f4870032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f487c0096a0 tx=0x7f487c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:29.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.113+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48780095f0 con 0x7f48880f51a0 2024-01-21T11:15:29.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.319+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f488810a0b0 con 0x7f48880f51a0 2024-01-21T11:15:29.321 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.320+0000 7f486f7fe700 1 -- 172.21.15.178:0/2969926401 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f487801cb10 con 0x7f48880f51a0 2024-01-21T11:15:29.321 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:29.321 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:29.322 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f487002ff60 msgr2=0x7f4870032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f487002ff60 0x7f4870032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f487c0096a0 tx=0x7f487c006b40).stop 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 msgr2=0x7f48881091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48881091e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4878013510 tx=0x7f487800b040).stop 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 shutdown_connections 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f487002ff60 0x7f4870032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 --2- 172.21.15.178:0/2969926401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880f51a0 0x7f48881091e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 >> 172.21.15.178:0/2969926401 conn(0x7f48880f09a0 msgr2=0x7f48880f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 shutdown_connections 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:29.322+0000 7f4887fff700 1 -- 172.21.15.178:0/2969926401 wait complete. 2024-01-21T11:15:29.323 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:29.683 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:29.321763+0000 mon.smithi089 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.178:0/2969926401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:30.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:30.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:30.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:29.658929+0000 mgr.smithi089.ufybqw (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:31.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3254327870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 msgr2=0x7fcb6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:31.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 --2- 172.21.15.178:0/3254327870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c0f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fcb60009fb0 tx=0x7fcb60009b30).stop 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3254327870 shutdown_connections 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 --2- 172.21.15.178:0/3254327870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c0f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3254327870 >> 172.21.15.178:0/3254327870 conn(0x7fcb6c0f1c70 msgr2=0x7fcb6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3254327870 shutdown_connections 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.852+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3254327870 wait complete. 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb71b0e700 1 Processor -- start 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb71b0e700 1 -- start start 2024-01-21T11:15:31.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb71b0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:31.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb71b0e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c069e20 con 0x7fcb6c0f6510 2024-01-21T11:15:31.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb6b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:31.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb6b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58214/0 (socket says 172.21.15.178:58214) 2024-01-21T11:15:31.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.853+0000 7fcb6b7fe700 1 -- 172.21.15.178:0/3114686567 learned_addr learned my addr 172.21.15.178:0/3114686567 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:31.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb6b7fe700 1 -- 172.21.15.178:0/3114686567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb60005040 con 0x7fcb6c0f6510 2024-01-21T11:15:31.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb6b7fe700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c06b9a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fcb60013510 tx=0x7fcb6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:31.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb60009320 con 0x7fcb6c0f6510 2024-01-21T11:15:31.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c069fc0 con 0x7fcb6c0f6510 2024-01-21T11:15:31.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb60013f00 con 0x7fcb6c0f6510 2024-01-21T11:15:31.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb60014d90 con 0x7fcb6c0f6510 2024-01-21T11:15:31.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.854+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c06a450 con 0x7fcb6c0f6510 2024-01-21T11:15:31.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.855+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb60014420 con 0x7fcb6c0f6510 2024-01-21T11:15:31.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.855+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c059100 con 0x7fcb6c0f6510 2024-01-21T11:15:31.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.855+0000 7fcb53fff700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcb5402ff60 0x7fcb54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:31.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.855+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb600410c0 con 0x7fcb6c0f6510 2024-01-21T11:15:31.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.855+0000 7fcb6affd700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcb5402ff60 0x7fcb54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:31.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.856+0000 7fcb6affd700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcb5402ff60 0x7fcb54032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fcb5c0096a0 tx=0x7fcb5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:31.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:31.860+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb600095f0 con 0x7fcb6c0f6510 2024-01-21T11:15:32.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.068+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb6c0fbf60 con 0x7fcb6c0f6510 2024-01-21T11:15:32.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.069+0000 7fcb53fff700 1 -- 172.21.15.178:0/3114686567 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb6001db70 con 0x7fcb6c0f6510 2024-01-21T11:15:32.070 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:32.071 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:32.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcb5402ff60 msgr2=0x7fcb54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcb5402ff60 0x7fcb54032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fcb5c0096a0 tx=0x7fcb5c006b40).stop 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 msgr2=0x7fcb6c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c06b9a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fcb60013510 tx=0x7fcb6000b040).stop 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 shutdown_connections 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcb5402ff60 0x7fcb54032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 --2- 172.21.15.178:0/3114686567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb6c0f6510 0x7fcb6c06b9a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:32.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 >> 172.21.15.178:0/3114686567 conn(0x7fcb6c0f1c70 msgr2=0x7fcb6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:32.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 shutdown_connections 2024-01-21T11:15:32.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:32.070+0000 7fcb71b0e700 1 -- 172.21.15.178:0/3114686567 wait complete. 2024-01-21T11:15:32.074 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:32.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:31.659222+0000 mgr.smithi089.ufybqw (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:32.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:32.070546+0000 mon.smithi089 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.178:0/3114686567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:33.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:33.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:34.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 -- 172.21.15.178:0/762270515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 msgr2=0x7f170c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 --2- 172.21.15.178:0/762270515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c0f66c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f16f4009fb0 tx=0x7f16f4009b30).stop 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 -- 172.21.15.178:0/762270515 shutdown_connections 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 --2- 172.21.15.178:0/762270515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c0f66c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 -- 172.21.15.178:0/762270515 >> 172.21.15.178:0/762270515 conn(0x7f170c0f1aa0 msgr2=0x7f170c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 -- 172.21.15.178:0/762270515 shutdown_connections 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.650+0000 7f17112b3700 1 -- 172.21.15.178:0/762270515 wait complete. 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.651+0000 7f17112b3700 1 Processor -- start 2024-01-21T11:15:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.651+0000 7f17112b3700 1 -- start start 2024-01-21T11:15:34.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.651+0000 7f17112b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.651+0000 7f17112b3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1069a0 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f170bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f170bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49168/0 (socket says 172.21.15.178:49168) 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f170bfff700 1 -- 172.21.15.178:0/957333054 learned_addr learned my addr 172.21.15.178:0/957333054 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f170bfff700 1 -- 172.21.15.178:0/957333054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f4005040 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f170bfff700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c1064b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f16f4013510 tx=0x7f16f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16f4009320 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c106ba0 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16f4013f00 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16f4014dc0 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.652+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c107030 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.653+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16f401d3d0 con 0x7f170c0f62a0 2024-01-21T11:15:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.653+0000 7f1708ff9700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f16f802ff60 0x7f16f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.653+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16f40410f0 con 0x7f170c0f62a0 2024-01-21T11:15:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.653+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c045ad0 con 0x7f170c0f62a0 2024-01-21T11:15:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.653+0000 7f170b7fe700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f16f802ff60 0x7f16f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.654+0000 7f170b7fe700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f16f802ff60 0x7f16f8032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f16fc0096a0 tx=0x7f16fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:34.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.658+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16f4012920 con 0x7f170c0f62a0 2024-01-21T11:15:34.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.861+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f170c107d50 con 0x7f170c0f62a0 2024-01-21T11:15:34.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.863+0000 7f1708ff9700 1 -- 172.21.15.178:0/957333054 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16f402b020 con 0x7f170c0f62a0 2024-01-21T11:15:34.863 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:34.863 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:34.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.863+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f16f802ff60 msgr2=0x7f16f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:34.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.863+0000 7f17112b3700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f16f802ff60 0x7f16f8032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f16fc0096a0 tx=0x7f16fc006b40).stop 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.863+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 msgr2=0x7f170c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.863+0000 7f17112b3700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c1064b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f16f4013510 tx=0x7f16f400b040).stop 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.864+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 shutdown_connections 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.864+0000 7f17112b3700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f16f802ff60 0x7f16f8032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.864+0000 7f17112b3700 1 --2- 172.21.15.178:0/957333054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f170c0f62a0 0x7f170c1064b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.864+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 >> 172.21.15.178:0/957333054 conn(0x7f170c0f1aa0 msgr2=0x7f170c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.864+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 shutdown_connections 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:34.864+0000 7f17112b3700 1 -- 172.21.15.178:0/957333054 wait complete. 2024-01-21T11:15:34.865 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:34.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:33.659562+0000 mgr.smithi089.ufybqw (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:35.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:34.863739+0000 mon.smithi089 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.178:0/957333054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:36.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:36.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:36.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:35.659812+0000 mgr.smithi089.ufybqw (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:37.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 -- 172.21.15.178:0/1749198010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 msgr2=0x7fae180f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:37.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 --2- 172.21.15.178:0/1749198010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae180f5d70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fae00009fb0 tx=0x7fae00009b30).stop 2024-01-21T11:15:37.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 -- 172.21.15.178:0/1749198010 shutdown_connections 2024-01-21T11:15:37.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 --2- 172.21.15.178:0/1749198010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae180f5d70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 -- 172.21.15.178:0/1749198010 >> 172.21.15.178:0/1749198010 conn(0x7fae180f0260 msgr2=0x7fae180f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 -- 172.21.15.178:0/1749198010 shutdown_connections 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.394+0000 7fae1e71a700 1 -- 172.21.15.178:0/1749198010 wait complete. 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.395+0000 7fae1e71a700 1 Processor -- start 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.395+0000 7fae1e71a700 1 -- start start 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.395+0000 7fae1e71a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae18105610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.395+0000 7fae1e71a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae18105b00 con 0x7fae180f5950 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae17fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae18105610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae17fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae18105610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49178/0 (socket says 172.21.15.178:49178) 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae17fff700 1 -- 172.21.15.178:0/752663917 learned_addr learned my addr 172.21.15.178:0/752663917 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae17fff700 1 -- 172.21.15.178:0/752663917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00005040 con 0x7fae180f5950 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae17fff700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae18105610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fae00013510 tx=0x7fae0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae00009320 con 0x7fae180f5950 2024-01-21T11:15:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae18105d00 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae00013f00 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae00014d90 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.396+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae18106190 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.397+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae000095f0 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.397+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae18045ad0 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.397+0000 7fae14ff9700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fae0402ff60 0x7fae04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.397+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae00041000 con 0x7fae180f5950 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.398+0000 7fae177fe700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fae0402ff60 0x7fae04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:37.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.398+0000 7fae177fe700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fae0402ff60 0x7fae04032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fae080096a0 tx=0x7fae08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:37.401 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.402+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae00014420 con 0x7fae180f5950 2024-01-21T11:15:37.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.608+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae18107040 con 0x7fae180f5950 2024-01-21T11:15:37.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.609+0000 7fae14ff9700 1 -- 172.21.15.178:0/752663917 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae00010230 con 0x7fae180f5950 2024-01-21T11:15:37.609 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:37.609 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.610+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fae0402ff60 msgr2=0x7fae04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.610+0000 7fae1e71a700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fae0402ff60 0x7fae04032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fae080096a0 tx=0x7fae08006b40).stop 2024-01-21T11:15:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.610+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 msgr2=0x7fae18105610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.610+0000 7fae1e71a700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae18105610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fae00013510 tx=0x7fae0000b040).stop 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.611+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 shutdown_connections 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.611+0000 7fae1e71a700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fae0402ff60 0x7fae04032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.611+0000 7fae1e71a700 1 --2- 172.21.15.178:0/752663917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae180f5950 0x7fae18105610 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.611+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 >> 172.21.15.178:0/752663917 conn(0x7fae180f0260 msgr2=0x7fae180f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.611+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 shutdown_connections 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:37.611+0000 7fae1e71a700 1 -- 172.21.15.178:0/752663917 wait complete. 2024-01-21T11:15:37.611 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:37.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:37 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:37.609795+0000 mon.smithi089 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.178:0/752663917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:38.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:37.660146+0000 mgr.smithi089.ufybqw (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:38.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:38.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:40.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2518688452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 msgr2=0x7f3cfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:40.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 --2- 172.21.15.178:0/2518688452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc0f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f3cec009fb0 tx=0x7f3cec009b30).stop 2024-01-21T11:15:40.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2518688452 shutdown_connections 2024-01-21T11:15:40.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 --2- 172.21.15.178:0/2518688452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc0f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:40.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2518688452 >> 172.21.15.178:0/2518688452 conn(0x7f3cfc0f0b70 msgr2=0x7f3cfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:40.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2518688452 shutdown_connections 2024-01-21T11:15:40.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.151+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2518688452 wait complete. 2024-01-21T11:15:40.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.152+0000 7f3d01fc6700 1 Processor -- start 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.152+0000 7f3d01fc6700 1 -- start start 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.152+0000 7f3d01fc6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.152+0000 7f3d01fc6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc060240 con 0x7f3cfc0f5410 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cfb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cfb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49196/0 (socket says 172.21.15.178:49196) 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cfb7fe700 1 -- 172.21.15.178:0/2084585866 learned_addr learned my addr 172.21.15.178:0/2084585866 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cfb7fe700 1 -- 172.21.15.178:0/2084585866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec005040 con 0x7f3cfc0f5410 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cfb7fe700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc109400 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3cec013510 tx=0x7f3cec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec009320 con 0x7f3cfc0f5410 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cec013f00 con 0x7f3cfc0f5410 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc060440 con 0x7f3cfc0f5410 2024-01-21T11:15:40.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec014d90 con 0x7f3cfc0f5410 2024-01-21T11:15:40.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.153+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc109ab0 con 0x7f3cfc0f5410 2024-01-21T11:15:40.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.154+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc045ad0 con 0x7f3cfc0f5410 2024-01-21T11:15:40.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.154+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cec0095f0 con 0x7f3cfc0f5410 2024-01-21T11:15:40.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.154+0000 7f3cf3fff700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3ce802ff60 0x7f3ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:40.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.154+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cec0410a0 con 0x7f3cfc0f5410 2024-01-21T11:15:40.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.155+0000 7f3cfaffd700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3ce802ff60 0x7f3ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:40.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.155+0000 7f3cfaffd700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3ce802ff60 0x7f3ce8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3ce40096a0 tx=0x7f3ce4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:40.159 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.159+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cec01d7f0 con 0x7f3cfc0f5410 2024-01-21T11:15:40.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.368+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cfc0fae60 con 0x7f3cfc0f5410 2024-01-21T11:15:40.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.368+0000 7f3cf3fff700 1 -- 172.21.15.178:0/2084585866 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cec010230 con 0x7f3cfc0f5410 2024-01-21T11:15:40.368 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:40.368 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:40.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3ce802ff60 msgr2=0x7f3ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:40.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3ce802ff60 0x7f3ce8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3ce40096a0 tx=0x7f3ce4006b40).stop 2024-01-21T11:15:40.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 msgr2=0x7f3cfc109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:40.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc109400 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3cec013510 tx=0x7f3cec00b040).stop 2024-01-21T11:15:40.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 shutdown_connections 2024-01-21T11:15:40.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3ce802ff60 0x7f3ce8032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:40.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 --2- 172.21.15.178:0/2084585866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cfc0f5410 0x7f3cfc109400 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:40.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 >> 172.21.15.178:0/2084585866 conn(0x7f3cfc0f0b70 msgr2=0x7f3cfc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:40.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 shutdown_connections 2024-01-21T11:15:40.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:40.370+0000 7f3d01fc6700 1 -- 172.21.15.178:0/2084585866 wait complete. 2024-01-21T11:15:40.371 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:40.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:39.660433+0000 mgr.smithi089.ufybqw (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:40.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:40.369011+0000 mon.smithi089 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.178:0/2084585866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:41.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:41.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 -- 172.21.15.178:0/612945899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 msgr2=0x7f2b380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 --2- 172.21.15.178:0/612945899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b380f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2b2c009fb0 tx=0x7f2b2c009b30).stop 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 -- 172.21.15.178:0/612945899 shutdown_connections 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 --2- 172.21.15.178:0/612945899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b380f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 -- 172.21.15.178:0/612945899 >> 172.21.15.178:0/612945899 conn(0x7f2b380f1c70 msgr2=0x7f2b380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 -- 172.21.15.178:0/612945899 shutdown_connections 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 -- 172.21.15.178:0/612945899 wait complete. 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.880+0000 7f2b3d20f700 1 Processor -- start 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b3d20f700 1 -- start start 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b3d20f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b3806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b3d20f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b38069e20 con 0x7f2b380f6510 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b36d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b3806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b36d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b3806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36952/0 (socket says 172.21.15.178:36952) 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b36d9d700 1 -- 172.21.15.178:0/3605947409 learned_addr learned my addr 172.21.15.178:0/3605947409 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b36d9d700 1 -- 172.21.15.178:0/3605947409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c005040 con 0x7f2b380f6510 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b36d9d700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b3806b9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2b2c013510 tx=0x7f2b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.881+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b2c009320 con 0x7f2b380f6510 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b38069fc0 con 0x7f2b380f6510 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b2c013f00 con 0x7f2b380f6510 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b2c014d90 con 0x7f2b380f6510 2024-01-21T11:15:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3806a450 con 0x7f2b380f6510 2024-01-21T11:15:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b2c014420 con 0x7f2b380f6510 2024-01-21T11:15:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b38059100 con 0x7f2b380f6510 2024-01-21T11:15:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.882+0000 7f2b1f7fe700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2b2002ff60 0x7f2b20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.883+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b2c041000 con 0x7f2b380f6510 2024-01-21T11:15:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.883+0000 7f2b3659c700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2b2002ff60 0x7f2b20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.883+0000 7f2b3659c700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2b2002ff60 0x7f2b20032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f2b280096a0 tx=0x7f2b28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:42.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:42.887+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b2c0095f0 con 0x7f2b380f6510 2024-01-21T11:15:42.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:41.660793+0000 mgr.smithi089.ufybqw (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.094+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b380fbf60 con 0x7f2b380f6510 2024-01-21T11:15:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.095+0000 7f2b1f7fe700 1 -- 172.21.15.178:0/3605947409 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b2c01db70 con 0x7f2b380f6510 2024-01-21T11:15:43.095 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:43.095 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:43.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.096+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2b2002ff60 msgr2=0x7f2b20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:43.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2b2002ff60 0x7f2b20032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f2b280096a0 tx=0x7f2b28006b40).stop 2024-01-21T11:15:43.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 msgr2=0x7f2b3806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b3806b9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2b2c013510 tx=0x7f2b2c00b040).stop 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 shutdown_connections 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2b2002ff60 0x7f2b20032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 --2- 172.21.15.178:0/3605947409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b380f6510 0x7f2b3806b9a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 >> 172.21.15.178:0/3605947409 conn(0x7f2b380f1c70 msgr2=0x7f2b380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 shutdown_connections 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:43.097+0000 7f2b3d20f700 1 -- 172.21.15.178:0/3605947409 wait complete. 2024-01-21T11:15:43.097 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:44.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:43.095508+0000 mon.smithi089 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.178:0/3605947409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:44.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:44.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:45.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:43.661139+0000 mgr.smithi089.ufybqw (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:45.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 -- 172.21.15.178:0/1433393939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 msgr2=0x7f42d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:45.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 --2- 172.21.15.178:0/1433393939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d80f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f42cc009fb0 tx=0x7f42cc009b30).stop 2024-01-21T11:15:45.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 -- 172.21.15.178:0/1433393939 shutdown_connections 2024-01-21T11:15:45.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 --2- 172.21.15.178:0/1433393939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d80f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:45.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 -- 172.21.15.178:0/1433393939 >> 172.21.15.178:0/1433393939 conn(0x7f42d80f1c70 msgr2=0x7f42d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:45.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 -- 172.21.15.178:0/1433393939 shutdown_connections 2024-01-21T11:15:45.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.666+0000 7f42ddc2b700 1 -- 172.21.15.178:0/1433393939 wait complete. 2024-01-21T11:15:45.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.667+0000 7f42ddc2b700 1 Processor -- start 2024-01-21T11:15:45.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.667+0000 7f42ddc2b700 1 -- start start 2024-01-21T11:15:45.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.667+0000 7f42ddc2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:45.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.667+0000 7f42ddc2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d8106bc0 con 0x7f42d80f6510 2024-01-21T11:15:45.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42d77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42d77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36970/0 (socket says 172.21.15.178:36970) 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42d77fe700 1 -- 172.21.15.178:0/2878790352 learned_addr learned my addr 172.21.15.178:0/2878790352 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42d77fe700 1 -- 172.21.15.178:0/2878790352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc005040 con 0x7f42d80f6510 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42d77fe700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d81066d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f42cc013510 tx=0x7f42cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42cc009320 con 0x7f42d80f6510 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42cc013f00 con 0x7f42d80f6510 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d8106dc0 con 0x7f42d80f6510 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42cc014d90 con 0x7f42d80f6510 2024-01-21T11:15:45.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.668+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d8107250 con 0x7f42d80f6510 2024-01-21T11:15:45.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.669+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42cc014420 con 0x7f42d80f6510 2024-01-21T11:15:45.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.669+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d8045ad0 con 0x7f42d80f6510 2024-01-21T11:15:45.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.669+0000 7f42bffff700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42c002ff60 0x7f42c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:45.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.669+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42cc041000 con 0x7f42d80f6510 2024-01-21T11:15:45.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.670+0000 7f42d6ffd700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42c002ff60 0x7f42c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:45.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.670+0000 7f42d6ffd700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42c002ff60 0x7f42c0032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f42c80096a0 tx=0x7f42c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:45.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.674+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42cc0095f0 con 0x7f42d80f6510 2024-01-21T11:15:45.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.880+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42d8108100 con 0x7f42d80f6510 2024-01-21T11:15:45.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.881+0000 7f42bffff700 1 -- 172.21.15.178:0/2878790352 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42cc01db70 con 0x7f42d80f6510 2024-01-21T11:15:45.881 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:45.881 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42c002ff60 msgr2=0x7f42c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42c002ff60 0x7f42c0032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f42c80096a0 tx=0x7f42c8006b40).stop 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 msgr2=0x7f42d81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d81066d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f42cc013510 tx=0x7f42cc00b040).stop 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 shutdown_connections 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42c002ff60 0x7f42c0032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 --2- 172.21.15.178:0/2878790352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42d80f6510 0x7f42d81066d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 >> 172.21.15.178:0/2878790352 conn(0x7f42d80f1c70 msgr2=0x7f42d80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 shutdown_connections 2024-01-21T11:15:45.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:45.883+0000 7f42ddc2b700 1 -- 172.21.15.178:0/2878790352 wait complete. 2024-01-21T11:15:45.884 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:47.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:45.661417+0000 mgr.smithi089.ufybqw (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:47.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:45.881767+0000 mon.smithi089 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.178:0/2878790352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:47.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:47.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:48.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 -- 172.21.15.178:0/1176784438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 msgr2=0x7fa1a40fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:48.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 --2- 172.21.15.178:0/1176784438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a40fb570 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa194009fb0 tx=0x7fa194009b30).stop 2024-01-21T11:15:48.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 -- 172.21.15.178:0/1176784438 shutdown_connections 2024-01-21T11:15:48.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 --2- 172.21.15.178:0/1176784438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a40fb570 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 -- 172.21.15.178:0/1176784438 >> 172.21.15.178:0/1176784438 conn(0x7fa1a40f0b70 msgr2=0x7fa1a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 -- 172.21.15.178:0/1176784438 shutdown_connections 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.510+0000 7fa1aa166700 1 -- 172.21.15.178:0/1176784438 wait complete. 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.511+0000 7fa1aa166700 1 Processor -- start 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.511+0000 7fa1aa166700 1 -- start start 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.511+0000 7fa1aa166700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a41088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.511+0000 7fa1aa166700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a4108dc0 con 0x7fa1a4060200 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1a37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a41088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1a37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a41088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36988/0 (socket says 172.21.15.178:36988) 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1a37fe700 1 -- 172.21.15.178:0/2399668162 learned_addr learned my addr 172.21.15.178:0/2399668162 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1a37fe700 1 -- 172.21.15.178:0/2399668162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa194005040 con 0x7fa1a4060200 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1a37fe700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a41088d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa194013510 tx=0x7fa19400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa194009320 con 0x7fa1a4060200 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a4108fc0 con 0x7fa1a4060200 2024-01-21T11:15:48.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa194013f00 con 0x7fa1a4060200 2024-01-21T11:15:48.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa194014dc0 con 0x7fa1a4060200 2024-01-21T11:15:48.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.512+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a4109450 con 0x7fa1a4060200 2024-01-21T11:15:48.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.513+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a4045ad0 con 0x7fa1a4060200 2024-01-21T11:15:48.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.513+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1940095f0 con 0x7fa1a4060200 2024-01-21T11:15:48.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.513+0000 7fa18bfff700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa18c02ff60 0x7fa18c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:48.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.513+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa194041020 con 0x7fa1a4060200 2024-01-21T11:15:48.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.514+0000 7fa1a2ffd700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa18c02ff60 0x7fa18c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:48.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.514+0000 7fa1a2ffd700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa18c02ff60 0x7fa18c032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa1980096a0 tx=0x7fa198006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:48.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.518+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa194014420 con 0x7fa1a4060200 2024-01-21T11:15:48.722 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.722+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1a410a2a0 con 0x7fa1a4060200 2024-01-21T11:15:48.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.723+0000 7fa18bfff700 1 -- 172.21.15.178:0/2399668162 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa19401c920 con 0x7fa1a4060200 2024-01-21T11:15:48.723 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:48.723 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:48.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa18c02ff60 msgr2=0x7fa18c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:48.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa18c02ff60 0x7fa18c032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa1980096a0 tx=0x7fa198006b40).stop 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 msgr2=0x7fa1a41088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a41088d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa194013510 tx=0x7fa19400b040).stop 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 shutdown_connections 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa18c02ff60 0x7fa18c032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 --2- 172.21.15.178:0/2399668162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1a4060200 0x7fa1a41088d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 >> 172.21.15.178:0/2399668162 conn(0x7fa1a40f0b70 msgr2=0x7fa1a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 shutdown_connections 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:48.725+0000 7fa1aa166700 1 -- 172.21.15.178:0/2399668162 wait complete. 2024-01-21T11:15:48.725 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:49.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:47.661696+0000 mgr.smithi089.ufybqw (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:50.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:50.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:50.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:49 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:48.723800+0000 mon.smithi089 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.178:0/2399668162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:50.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:49 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:49.661992+0000 mgr.smithi089.ufybqw (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 -- 172.21.15.178:0/1307055536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 msgr2=0x7f7bf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 --2- 172.21.15.178:0/1307055536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf00f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7bd8009fb0 tx=0x7f7bd8009b30).stop 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 -- 172.21.15.178:0/1307055536 shutdown_connections 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 --2- 172.21.15.178:0/1307055536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf00f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 -- 172.21.15.178:0/1307055536 >> 172.21.15.178:0/1307055536 conn(0x7f7bf00f0b70 msgr2=0x7f7bf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 -- 172.21.15.178:0/1307055536 shutdown_connections 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.228+0000 7f7bf4924700 1 -- 172.21.15.178:0/1307055536 wait complete. 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bf4924700 1 Processor -- start 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bf4924700 1 -- start start 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bf4924700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bf4924700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0060240 con 0x7f7bf00f5410 2024-01-21T11:15:51.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:37006/0 (socket says 172.21.15.178:37006) 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bee59c700 1 -- 172.21.15.178:0/866541506 learned_addr learned my addr 172.21.15.178:0/866541506 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.229+0000 7f7bee59c700 1 -- 172.21.15.178:0/866541506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8005040 con 0x7f7bf00f5410 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.230+0000 7f7bee59c700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf0109430 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7bd8013510 tx=0x7f7bd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.230+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bd8009320 con 0x7f7bf00f5410 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.230+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bd8013f00 con 0x7f7bf00f5410 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.230+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf0060440 con 0x7f7bf00f5410 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.230+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bd8014d90 con 0x7f7bf00f5410 2024-01-21T11:15:51.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.230+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf0109ae0 con 0x7f7bf00f5410 2024-01-21T11:15:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.231+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bd80095f0 con 0x7f7bf00f5410 2024-01-21T11:15:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.231+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0045ad0 con 0x7f7bf00f5410 2024-01-21T11:15:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.231+0000 7f7be6ffd700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7bdc02ff60 0x7f7bdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.231+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bd8040fe0 con 0x7f7bf00f5410 2024-01-21T11:15:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.231+0000 7f7bedd9b700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7bdc02ff60 0x7f7bdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.231+0000 7f7bedd9b700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7bdc02ff60 0x7f7bdc032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f7be00096a0 tx=0x7f7be0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:51.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.235+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bd801d7f0 con 0x7f7bf00f5410 2024-01-21T11:15:51.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.441+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bf00fae60 con 0x7f7bf00f5410 2024-01-21T11:15:51.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.442+0000 7f7be6ffd700 1 -- 172.21.15.178:0/866541506 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bd8010230 con 0x7f7bf00f5410 2024-01-21T11:15:51.442 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:51.442 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:51.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.443+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7bdc02ff60 msgr2=0x7f7bdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:51.443 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.443+0000 7f7bf4924700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7bdc02ff60 0x7f7bdc032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f7be00096a0 tx=0x7f7be0006b40).stop 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 msgr2=0x7f7bf0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf0109430 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7bd8013510 tx=0x7f7bd800b040).stop 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 shutdown_connections 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7bdc02ff60 0x7f7bdc032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 --2- 172.21.15.178:0/866541506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf00f5410 0x7f7bf0109430 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 >> 172.21.15.178:0/866541506 conn(0x7f7bf00f0b70 msgr2=0x7f7bf00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 shutdown_connections 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:51.444+0000 7f7bf4924700 1 -- 172.21.15.178:0/866541506 wait complete. 2024-01-21T11:15:51.444 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:51.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:51 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:51.442726+0000 mon.smithi089 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.178:0/866541506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:52.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:52.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:52.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:51.662232+0000 mgr.smithi089.ufybqw (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:54.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1524683726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 msgr2=0x7f0ad00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:54.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 --2- 172.21.15.178:0/1524683726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad00f90f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f0ab8009fb0 tx=0x7f0ab8009b30).stop 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1524683726 shutdown_connections 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 --2- 172.21.15.178:0/1524683726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad00f90f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1524683726 >> 172.21.15.178:0/1524683726 conn(0x7f0ad00f0b70 msgr2=0x7f0ad00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1524683726 shutdown_connections 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1524683726 wait complete. 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.018+0000 7f0ad60ed700 1 Processor -- start 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0ad60ed700 1 -- start start 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0ad60ed700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:54.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0ad60ed700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad0106bd0 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0acf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0acf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34280/0 (socket says 172.21.15.178:34280) 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0acf7fe700 1 -- 172.21.15.178:0/1065413511 learned_addr learned my addr 172.21.15.178:0/1065413511 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0acf7fe700 1 -- 172.21.15.178:0/1065413511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab8005040 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.019+0000 7f0acf7fe700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad01066e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0ab8013510 tx=0x7f0ab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.020+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab8009320 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.020+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab8013f00 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.020+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad0106dd0 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.020+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab8014dc0 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.020+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad0107260 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.021+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ab80095f0 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.021+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad0045ad0 con 0x7f0ad0060280 2024-01-21T11:15:54.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.021+0000 7f0ac7fff700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0abc02ff60 0x7f0abc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:54.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.021+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ab8041000 con 0x7f0ad0060280 2024-01-21T11:15:54.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.021+0000 7f0aceffd700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0abc02ff60 0x7f0abc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:54.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.022+0000 7f0aceffd700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0abc02ff60 0x7f0abc032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0ac00096a0 tx=0x7f0ac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:54.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.026+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ab8014420 con 0x7f0ad0060280 2024-01-21T11:15:54.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:53.662552+0000 mgr.smithi089.ufybqw (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:54.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.229+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ad0108110 con 0x7f0ad0060280 2024-01-21T11:15:54.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.230+0000 7f0ac7fff700 1 -- 172.21.15.178:0/1065413511 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ab8010230 con 0x7f0ad0060280 2024-01-21T11:15:54.230 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:54.230 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0abc02ff60 msgr2=0x7f0abc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0abc02ff60 0x7f0abc032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0ac00096a0 tx=0x7f0ac0006b40).stop 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 msgr2=0x7f0ad01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad01066e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0ab8013510 tx=0x7f0ab800b040).stop 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 shutdown_connections 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0abc02ff60 0x7f0abc032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 --2- 172.21.15.178:0/1065413511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad0060280 0x7f0ad01066e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 >> 172.21.15.178:0/1065413511 conn(0x7f0ad00f0b70 msgr2=0x7f0ad00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 shutdown_connections 2024-01-21T11:15:54.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:54.232+0000 7f0ad60ed700 1 -- 172.21.15.178:0/1065413511 wait complete. 2024-01-21T11:15:54.233 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:55.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:54.230923+0000 mon.smithi089 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.178:0/1065413511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:55.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:55.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:56.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:55.662800+0000 mgr.smithi089.ufybqw (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:56.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 -- 172.21.15.178:0/3213956157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 msgr2=0x7fac640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:56.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 --2- 172.21.15.178:0/3213956157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac640f90f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fac58009fb0 tx=0x7fac58009b30).stop 2024-01-21T11:15:56.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 -- 172.21.15.178:0/3213956157 shutdown_connections 2024-01-21T11:15:56.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 --2- 172.21.15.178:0/3213956157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac640f90f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:56.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 -- 172.21.15.178:0/3213956157 >> 172.21.15.178:0/3213956157 conn(0x7fac640f0b70 msgr2=0x7fac640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:56.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 -- 172.21.15.178:0/3213956157 shutdown_connections 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.806+0000 7fac6a98a700 1 -- 172.21.15.178:0/3213956157 wait complete. 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac6a98a700 1 Processor -- start 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac6a98a700 1 -- start start 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac6a98a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac6a98a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac64106bd0 con 0x7fac64060280 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac63fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac63fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34306/0 (socket says 172.21.15.178:34306) 2024-01-21T11:15:56.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.807+0000 7fac63fff700 1 -- 172.21.15.178:0/422229570 learned_addr learned my addr 172.21.15.178:0/422229570 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:56.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac63fff700 1 -- 172.21.15.178:0/422229570 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58005040 con 0x7fac64060280 2024-01-21T11:15:56.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac63fff700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac641066e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fac58013510 tx=0x7fac5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:56.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac58009320 con 0x7fac64060280 2024-01-21T11:15:56.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac64106dd0 con 0x7fac64060280 2024-01-21T11:15:56.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac58013f00 con 0x7fac64060280 2024-01-21T11:15:56.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac58014dc0 con 0x7fac64060280 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.808+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac64107260 con 0x7fac64060280 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.809+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac5801d3d0 con 0x7fac64060280 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.809+0000 7fac60ff9700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fac4c02ff60 0x7fac4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.809+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac58041360 con 0x7fac64060280 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.809+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64045ad0 con 0x7fac64060280 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.809+0000 7fac637fe700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fac4c02ff60 0x7fac4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:56.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.810+0000 7fac637fe700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fac4c02ff60 0x7fac4c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fac540096a0 tx=0x7fac54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:56.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:56.814+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac58012920 con 0x7fac64060280 2024-01-21T11:15:57.017 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.017+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac64107ee0 con 0x7fac64060280 2024-01-21T11:15:57.017 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.017+0000 7fac60ff9700 1 -- 172.21.15.178:0/422229570 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac5802b030 con 0x7fac64060280 2024-01-21T11:15:57.017 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:57.017 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fac4c02ff60 msgr2=0x7fac4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fac4c02ff60 0x7fac4c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fac540096a0 tx=0x7fac54006b40).stop 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 msgr2=0x7fac641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac641066e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fac58013510 tx=0x7fac5800b040).stop 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 shutdown_connections 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fac4c02ff60 0x7fac4c032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 --2- 172.21.15.178:0/422229570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac64060280 0x7fac641066e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:57.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 >> 172.21.15.178:0/422229570 conn(0x7fac640f0b70 msgr2=0x7fac640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:57.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 shutdown_connections 2024-01-21T11:15:57.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:57.019+0000 7fac6a98a700 1 -- 172.21.15.178:0/422229570 wait complete. 2024-01-21T11:15:57.020 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:15:57.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:57 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:57.018055+0000 mon.smithi089 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.178:0/422229570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:15:58.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:15:58.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:15:58.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:15:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:57.663112+0000 mgr.smithi089.ufybqw (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:15:59.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 -- 172.21.15.178:0/2635223131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 msgr2=0x7fef400f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 --2- 172.21.15.178:0/2635223131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef400f8f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fef3c009fb0 tx=0x7fef3c009b30).stop 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 -- 172.21.15.178:0/2635223131 shutdown_connections 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 --2- 172.21.15.178:0/2635223131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef400f8f20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 -- 172.21.15.178:0/2635223131 >> 172.21.15.178:0/2635223131 conn(0x7fef400f09a0 msgr2=0x7fef400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 -- 172.21.15.178:0/2635223131 shutdown_connections 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 -- 172.21.15.178:0/2635223131 wait complete. 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.572+0000 7fef462f9700 1 Processor -- start 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef462f9700 1 -- start start 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef462f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef40106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef462f9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef401069f0 con 0x7fef400601e0 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef452f7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef40106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef452f7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef40106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:34322/0 (socket says 172.21.15.178:34322) 2024-01-21T11:15:59.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef452f7700 1 -- 172.21.15.178:0/3625474857 learned_addr learned my addr 172.21.15.178:0/3625474857 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef452f7700 1 -- 172.21.15.178:0/3625474857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c005040 con 0x7fef400601e0 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef452f7700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef40106500 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fef3c013510 tx=0x7fef3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.573+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c009320 con 0x7fef400601e0 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.574+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef40106bf0 con 0x7fef400601e0 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.574+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef3c013f00 con 0x7fef400601e0 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.574+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c014d90 con 0x7fef400601e0 2024-01-21T11:15:59.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.574+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef40107080 con 0x7fef400601e0 2024-01-21T11:15:59.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.574+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef40045ad0 con 0x7fef400601e0 2024-01-21T11:15:59.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.575+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef3c0095f0 con 0x7fef400601e0 2024-01-21T11:15:59.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.575+0000 7fef35ffb700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fef2c02ff60 0x7fef2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:15:59.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.575+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef3c040fe0 con 0x7fef400601e0 2024-01-21T11:15:59.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.575+0000 7fef44af6700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fef2c02ff60 0x7fef2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:15:59.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.576+0000 7fef44af6700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fef2c02ff60 0x7fef2c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fef300096a0 tx=0x7fef30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:15:59.579 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.579+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef3c014420 con 0x7fef400601e0 2024-01-21T11:15:59.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.785+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef40107ed0 con 0x7fef400601e0 2024-01-21T11:15:59.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.785+0000 7fef35ffb700 1 -- 172.21.15.178:0/3625474857 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef3c010230 con 0x7fef400601e0 2024-01-21T11:15:59.786 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:15:59.786 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fef2c02ff60 msgr2=0x7fef2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fef2c02ff60 0x7fef2c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fef300096a0 tx=0x7fef30006b40).stop 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 msgr2=0x7fef40106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef40106500 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fef3c013510 tx=0x7fef3c00b040).stop 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 shutdown_connections 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fef2c02ff60 0x7fef2c032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:59.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 --2- 172.21.15.178:0/3625474857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef400601e0 0x7fef40106500 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:15:59.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 >> 172.21.15.178:0/3625474857 conn(0x7fef400f09a0 msgr2=0x7fef400f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:15:59.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 shutdown_connections 2024-01-21T11:15:59.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:15:59.787+0000 7fef462f9700 1 -- 172.21.15.178:0/3625474857 wait complete. 2024-01-21T11:15:59.788 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:00.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:15:59.663358+0000 mgr.smithi089.ufybqw (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:00.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:15:59.786077+0000 mon.smithi089 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.178:0/3625474857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:01.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:01.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:02.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 -- 172.21.15.178:0/115675749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 msgr2=0x7f6bd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:02.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 --2- 172.21.15.178:0/115675749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd40f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f6bc4009fb0 tx=0x7f6bc4009b30).stop 2024-01-21T11:16:02.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 -- 172.21.15.178:0/115675749 shutdown_connections 2024-01-21T11:16:02.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 --2- 172.21.15.178:0/115675749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd40f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:02.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 -- 172.21.15.178:0/115675749 >> 172.21.15.178:0/115675749 conn(0x7f6bd40f1c70 msgr2=0x7f6bd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 -- 172.21.15.178:0/115675749 shutdown_connections 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 -- 172.21.15.178:0/115675749 wait complete. 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.415+0000 7f6bda779700 1 Processor -- start 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bda779700 1 -- start start 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bda779700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bda779700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd4069e20 con 0x7f6bd40f6510 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53908/0 (socket says 172.21.15.178:53908) 2024-01-21T11:16:02.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bd3fff700 1 -- 172.21.15.178:0/3801964332 learned_addr learned my addr 172.21.15.178:0/3801964332 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.416+0000 7f6bd3fff700 1 -- 172.21.15.178:0/3801964332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc4005040 con 0x7f6bd40f6510 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bd3fff700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd406b9a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6bc4013510 tx=0x7f6bc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bc4009320 con 0x7f6bd40f6510 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd4069fc0 con 0x7f6bd40f6510 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd406a450 con 0x7f6bd40f6510 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bc4013f00 con 0x7f6bd40f6510 2024-01-21T11:16:02.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bc4014d60 con 0x7f6bd40f6510 2024-01-21T11:16:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.417+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bc40263a0 con 0x7f6bd40f6510 2024-01-21T11:16:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.418+0000 7f6bd0ff9700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6bbc02ff60 0x7f6bbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.418+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bc4041040 con 0x7f6bd40f6510 2024-01-21T11:16:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.418+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd4059100 con 0x7f6bd40f6510 2024-01-21T11:16:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.418+0000 7f6bd37fe700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6bbc02ff60 0x7f6bbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.418+0000 7f6bd37fe700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6bbc02ff60 0x7f6bbc032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6bc80096a0 tx=0x7f6bc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:02.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.422+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bc401d380 con 0x7f6bd40f6510 2024-01-21T11:16:02.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.628+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bd40fbf60 con 0x7f6bd40f6510 2024-01-21T11:16:02.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.629+0000 7f6bd0ff9700 1 -- 172.21.15.178:0/3801964332 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6bc402b020 con 0x7f6bd40f6510 2024-01-21T11:16:02.628 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:02.629 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6bbc02ff60 msgr2=0x7f6bbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6bbc02ff60 0x7f6bbc032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6bc80096a0 tx=0x7f6bc8006b40).stop 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 msgr2=0x7f6bd406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd406b9a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6bc4013510 tx=0x7f6bc400b040).stop 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 shutdown_connections 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6bbc02ff60 0x7f6bbc032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 --2- 172.21.15.178:0/3801964332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bd40f6510 0x7f6bd406b9a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:02.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 >> 172.21.15.178:0/3801964332 conn(0x7f6bd40f1c70 msgr2=0x7f6bd40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:02.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 shutdown_connections 2024-01-21T11:16:02.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:02.630+0000 7f6bda779700 1 -- 172.21.15.178:0/3801964332 wait complete. 2024-01-21T11:16:02.631 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:02.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:01.663675+0000 mgr.smithi089.ufybqw (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:02.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:02.629287+0000 mon.smithi089 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.178:0/3801964332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:04.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:04.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:04.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:03.664046+0000 mgr.smithi089.ufybqw (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:05.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4023414109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 msgr2=0x7f77ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:05.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 --2- 172.21.15.178:0/4023414109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec0f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f77e0009fb0 tx=0x7f77e0009b30).stop 2024-01-21T11:16:05.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4023414109 shutdown_connections 2024-01-21T11:16:05.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 --2- 172.21.15.178:0/4023414109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec0f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:05.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4023414109 >> 172.21.15.178:0/4023414109 conn(0x7f77ec0f0b70 msgr2=0x7f77ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:05.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4023414109 shutdown_connections 2024-01-21T11:16:05.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.208+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4023414109 wait complete. 2024-01-21T11:16:05.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f2e8a700 1 Processor -- start 2024-01-21T11:16:05.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f2e8a700 1 -- start start 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f2e8a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f2e8a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec060240 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f0c26700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f0c26700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53932/0 (socket says 172.21.15.178:53932) 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.209+0000 7f77f0c26700 1 -- 172.21.15.178:0/4006668229 learned_addr learned my addr 172.21.15.178:0/4006668229 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77f0c26700 1 -- 172.21.15.178:0/4006668229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e0005040 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77f0c26700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f77e0013510 tx=0x7f77e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e0009320 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec060440 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77e0013f00 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e0014d90 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.210+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec109ae0 con 0x7f77ec0f5410 2024-01-21T11:16:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.211+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77e00263a0 con 0x7f77ec0f5410 2024-01-21T11:16:05.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.211+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec045ad0 con 0x7f77ec0f5410 2024-01-21T11:16:05.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.211+0000 7f77e97fa700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77dc02ff60 0x7f77dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:05.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.211+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77e0040ff0 con 0x7f77ec0f5410 2024-01-21T11:16:05.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.211+0000 7f77ebfff700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77dc02ff60 0x7f77dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:05.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.212+0000 7f77ebfff700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77dc02ff60 0x7f77dc032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f77d80096a0 tx=0x7f77d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:05.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.216+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77e001db70 con 0x7f77ec0f5410 2024-01-21T11:16:05.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.425+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77ec0fae60 con 0x7f77ec0f5410 2024-01-21T11:16:05.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.426+0000 7f77e97fa700 1 -- 172.21.15.178:0/4006668229 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77e001db70 con 0x7f77ec0f5410 2024-01-21T11:16:05.426 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:05.426 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:05.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.427+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77dc02ff60 msgr2=0x7f77dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77dc02ff60 0x7f77dc032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f77d80096a0 tx=0x7f77d8006b40).stop 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 msgr2=0x7f77ec109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f77e0013510 tx=0x7f77e000b040).stop 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 shutdown_connections 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77dc02ff60 0x7f77dc032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 --2- 172.21.15.178:0/4006668229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ec0f5410 0x7f77ec109430 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 >> 172.21.15.178:0/4006668229 conn(0x7f77ec0f0b70 msgr2=0x7f77ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 shutdown_connections 2024-01-21T11:16:05.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:05.428+0000 7f77f2e8a700 1 -- 172.21.15.178:0/4006668229 wait complete. 2024-01-21T11:16:05.429 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:05.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:05.426840+0000 mon.smithi089 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.178:0/4006668229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:06.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:06.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:06.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:05.664342+0000 mgr.smithi089.ufybqw (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:07.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 -- 172.21.15.178:0/411140056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 msgr2=0x7fdce80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 --2- 172.21.15.178:0/411140056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce80f90f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fdcd0009fb0 tx=0x7fdcd0009b30).stop 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 -- 172.21.15.178:0/411140056 shutdown_connections 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 --2- 172.21.15.178:0/411140056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce80f90f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 -- 172.21.15.178:0/411140056 >> 172.21.15.178:0/411140056 conn(0x7fdce80f0b70 msgr2=0x7fdce80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 -- 172.21.15.178:0/411140056 shutdown_connections 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 -- 172.21.15.178:0/411140056 wait complete. 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.990+0000 7fdcee0cb700 1 Processor -- start 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdcee0cb700 1 -- start start 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdcee0cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce81088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdcee0cb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8108de0 con 0x7fdce8060280 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdce77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce81088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdce77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce81088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53954/0 (socket says 172.21.15.178:53954) 2024-01-21T11:16:07.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdce77fe700 1 -- 172.21.15.178:0/2082958981 learned_addr learned my addr 172.21.15.178:0/2082958981 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdce77fe700 1 -- 172.21.15.178:0/2082958981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd0005040 con 0x7fdce8060280 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.991+0000 7fdce77fe700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce81088f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdcd0013510 tx=0x7fdcd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.992+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcd0009320 con 0x7fdce8060280 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.992+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce8108fe0 con 0x7fdce8060280 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.992+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcd0013f00 con 0x7fdce8060280 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.992+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce8109470 con 0x7fdce8060280 2024-01-21T11:16:07.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.992+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcd0014de0 con 0x7fdce8060280 2024-01-21T11:16:07.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.992+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcd0014420 con 0x7fdce8060280 2024-01-21T11:16:07.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.993+0000 7fdcdffff700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdcd402ff60 0x7fdcd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:07.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.993+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcd0041160 con 0x7fdce8060280 2024-01-21T11:16:07.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.993+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce8045ad0 con 0x7fdce8060280 2024-01-21T11:16:07.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.993+0000 7fdce6ffd700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdcd402ff60 0x7fdcd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:07.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.993+0000 7fdce6ffd700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdcd402ff60 0x7fdcd4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdcd80096a0 tx=0x7fdcd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:07.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:07.997+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcd00095f0 con 0x7fdce8060280 2024-01-21T11:16:08.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.205+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdce80fbf50 con 0x7fdce8060280 2024-01-21T11:16:08.207 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.206+0000 7fdcdffff700 1 -- 172.21.15.178:0/2082958981 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcd001cb10 con 0x7fdce8060280 2024-01-21T11:16:08.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:08.207 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:08.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.207+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdcd402ff60 msgr2=0x7fdcd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.207+0000 7fdcee0cb700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdcd402ff60 0x7fdcd4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdcd80096a0 tx=0x7fdcd8006b40).stop 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.207+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 msgr2=0x7fdce81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce81088f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdcd0013510 tx=0x7fdcd000b040).stop 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 shutdown_connections 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdcd402ff60 0x7fdcd4032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 --2- 172.21.15.178:0/2082958981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce8060280 0x7fdce81088f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 >> 172.21.15.178:0/2082958981 conn(0x7fdce80f0b70 msgr2=0x7fdce80f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 shutdown_connections 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:08.208+0000 7fdcee0cb700 1 -- 172.21.15.178:0/2082958981 wait complete. 2024-01-21T11:16:08.209 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:08.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:07.664613+0000 mgr.smithi089.ufybqw (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:08.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:08.207882+0000 mon.smithi089 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.178:0/2082958981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:09.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:09.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:10.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 -- 172.21.15.178:0/781990945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 msgr2=0x7fae040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:10.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 --2- 172.21.15.178:0/781990945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae040f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fadf4009fb0 tx=0x7fadf4009b30).stop 2024-01-21T11:16:10.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 -- 172.21.15.178:0/781990945 shutdown_connections 2024-01-21T11:16:10.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 --2- 172.21.15.178:0/781990945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae040f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:10.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 -- 172.21.15.178:0/781990945 >> 172.21.15.178:0/781990945 conn(0x7fae040f0b70 msgr2=0x7fae040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:10.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 -- 172.21.15.178:0/781990945 shutdown_connections 2024-01-21T11:16:10.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.797+0000 7fae0b8f2700 1 -- 172.21.15.178:0/781990945 wait complete. 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.798+0000 7fae0b8f2700 1 Processor -- start 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.798+0000 7fae0b8f2700 1 -- start start 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.798+0000 7fae0b8f2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae041093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.798+0000 7fae0b8f2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04060220 con 0x7fae040f5410 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0968e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae041093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0968e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae041093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53972/0 (socket says 172.21.15.178:53972) 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0968e700 1 -- 172.21.15.178:0/3615304921 learned_addr learned my addr 172.21.15.178:0/3615304921 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:10.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0968e700 1 -- 172.21.15.178:0/3615304921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4005040 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0968e700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae041093f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fadf4013510 tx=0x7fadf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf4009320 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadf4013f00 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae04060420 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf4014d90 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.799+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae04109aa0 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.800+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae04059100 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.800+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fadf40095f0 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.800+0000 7fadf9ffb700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fadf002ff60 0x7fadf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.800+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fadf4041000 con 0x7fae040f5410 2024-01-21T11:16:10.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.801+0000 7fae08e8d700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fadf002ff60 0x7fadf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:10.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.801+0000 7fae08e8d700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fadf002ff60 0x7fadf0032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fae000096a0 tx=0x7fae00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:10.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:10.805+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadf4014420 con 0x7fae040f5410 2024-01-21T11:16:10.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:09.664857+0000 mgr.smithi089.ufybqw (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:11.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.011+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae04045ad0 con 0x7fae040f5410 2024-01-21T11:16:11.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.012+0000 7fadf9ffb700 1 -- 172.21.15.178:0/3615304921 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fadf401d7f0 con 0x7fae040f5410 2024-01-21T11:16:11.013 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:11.013 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:11.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fadf002ff60 msgr2=0x7fadf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fadf002ff60 0x7fadf0032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fae000096a0 tx=0x7fae00006b40).stop 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 msgr2=0x7fae041093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae041093f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fadf4013510 tx=0x7fadf400b040).stop 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 shutdown_connections 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fadf002ff60 0x7fadf0032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 --2- 172.21.15.178:0/3615304921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae040f5410 0x7fae041093f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 >> 172.21.15.178:0/3615304921 conn(0x7fae040f0b70 msgr2=0x7fae040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 shutdown_connections 2024-01-21T11:16:11.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:11.014+0000 7fae0b8f2700 1 -- 172.21.15.178:0/3615304921 wait complete. 2024-01-21T11:16:11.016 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:11.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:11 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:11.013631+0000 mon.smithi089 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.178:0/3615304921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:12.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:12.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:12.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:11.665108+0000 mgr.smithi089.ufybqw (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:13.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 -- 172.21.15.178:0/2029106955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 msgr2=0x7f5fc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:13.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 --2- 172.21.15.178:0/2029106955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5fb0009fb0 tx=0x7f5fb0009b30).stop 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 -- 172.21.15.178:0/2029106955 shutdown_connections 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 --2- 172.21.15.178:0/2029106955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 -- 172.21.15.178:0/2029106955 >> 172.21.15.178:0/2029106955 conn(0x7f5fc00f0b70 msgr2=0x7f5fc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 -- 172.21.15.178:0/2029106955 shutdown_connections 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.543+0000 7f5fc6313700 1 -- 172.21.15.178:0/2029106955 wait complete. 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fc6313700 1 Processor -- start 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fc6313700 1 -- start start 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fc6313700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fc6313700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00fb820 con 0x7f5fc00f5410 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fbffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fbffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55018/0 (socket says 172.21.15.178:55018) 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.544+0000 7f5fbffff700 1 -- 172.21.15.178:0/1358423742 learned_addr learned my addr 172.21.15.178:0/1358423742 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:13.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fbffff700 1 -- 172.21.15.178:0/1358423742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb0005040 con 0x7f5fc00f5410 2024-01-21T11:16:13.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fbffff700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00fd3a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5fb0009f80 tx=0x7f5fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fb0009320 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc00fb9c0 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fb0013ec0 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fb001ece0 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.545+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc00fbe50 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.546+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fb001e420 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.546+0000 7f5fbcff9700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5fa802ff60 0x7f5fa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.546+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fb0040ed0 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.546+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0059100 con 0x7f5fc00f5410 2024-01-21T11:16:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.546+0000 7f5fbf7fe700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5fa802ff60 0x7f5fa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:13.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.546+0000 7f5fbf7fe700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5fa802ff60 0x7f5fa8032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5fb40096a0 tx=0x7f5fb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:13.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.550+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fb001e6f0 con 0x7f5fc00f5410 2024-01-21T11:16:13.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.758+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fc00fc550 con 0x7f5fc00f5410 2024-01-21T11:16:13.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.758+0000 7f5fbcff9700 1 -- 172.21.15.178:0/1358423742 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5fb001cb10 con 0x7f5fc00f5410 2024-01-21T11:16:13.759 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:13.759 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:13.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5fa802ff60 msgr2=0x7f5fa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5fa802ff60 0x7f5fa8032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5fb40096a0 tx=0x7f5fb4006b40).stop 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 msgr2=0x7f5fc00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00fd3a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5fb0009f80 tx=0x7f5fb000b040).stop 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 shutdown_connections 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5fa802ff60 0x7f5fa8032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 --2- 172.21.15.178:0/1358423742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fc00f5410 0x7f5fc00fd3a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 >> 172.21.15.178:0/1358423742 conn(0x7f5fc00f0b70 msgr2=0x7f5fc005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 shutdown_connections 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:13.760+0000 7f5fc6313700 1 -- 172.21.15.178:0/1358423742 wait complete. 2024-01-21T11:16:13.761 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:13.665367+0000 mgr.smithi089.ufybqw (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:13.759961+0000 mon.smithi089 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.178:0/1358423742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:13.773245+0000 mon.smithi089 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:16:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:13.782515+0000 mon.smithi089 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:16:15.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:15.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:16.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.330+0000 7f4b5096b700 1 -- 172.21.15.178:0/3808634906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 msgr2=0x7f4b4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:16.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.330+0000 7f4b5096b700 1 --2- 172.21.15.178:0/3808634906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c0f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4b40009fb0 tx=0x7f4b40009b30).stop 2024-01-21T11:16:16.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.330+0000 7f4b5096b700 1 -- 172.21.15.178:0/3808634906 shutdown_connections 2024-01-21T11:16:16.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.330+0000 7f4b5096b700 1 --2- 172.21.15.178:0/3808634906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c0f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.330+0000 7f4b5096b700 1 -- 172.21.15.178:0/3808634906 >> 172.21.15.178:0/3808634906 conn(0x7f4b4c0f0b70 msgr2=0x7f4b4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b5096b700 1 -- 172.21.15.178:0/3808634906 shutdown_connections 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b5096b700 1 -- 172.21.15.178:0/3808634906 wait complete. 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b5096b700 1 Processor -- start 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b5096b700 1 -- start start 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b5096b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b5096b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c106c10 con 0x7f4b4c0f5410 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b4a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b4a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55048/0 (socket says 172.21.15.178:55048) 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.331+0000 7f4b4a59c700 1 -- 172.21.15.178:0/107605804 learned_addr learned my addr 172.21.15.178:0/107605804 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b4a59c700 1 -- 172.21.15.178:0/107605804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40005040 con 0x7f4b4c0f5410 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b4a59c700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c106720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4b40013510 tx=0x7f4b4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b40009320 con 0x7f4b4c0f5410 2024-01-21T11:16:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c106e10 con 0x7f4b4c0f5410 2024-01-21T11:16:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b40013f00 con 0x7f4b4c0f5410 2024-01-21T11:16:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b40014d90 con 0x7f4b4c0f5410 2024-01-21T11:16:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.332+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c1072a0 con 0x7f4b4c0f5410 2024-01-21T11:16:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.333+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b40014420 con 0x7f4b4c0f5410 2024-01-21T11:16:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.333+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4c045ad0 con 0x7f4b4c0f5410 2024-01-21T11:16:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.333+0000 7f4b3affd700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4b3402ff60 0x7f4b34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.333+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b40040ff0 con 0x7f4b4c0f5410 2024-01-21T11:16:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.333+0000 7f4b49d9b700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4b3402ff60 0x7f4b34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.334+0000 7f4b49d9b700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4b3402ff60 0x7f4b34032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4b3c0096a0 tx=0x7f4b3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:16.338 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.338+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b400095f0 con 0x7f4b4c0f5410 2024-01-21T11:16:16.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.544+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b4c108150 con 0x7f4b4c0f5410 2024-01-21T11:16:16.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.545+0000 7f4b3affd700 1 -- 172.21.15.178:0/107605804 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b4001db70 con 0x7f4b4c0f5410 2024-01-21T11:16:16.546 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:16.546 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:16.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4b3402ff60 msgr2=0x7f4b34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:16.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4b3402ff60 0x7f4b34032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4b3c0096a0 tx=0x7f4b3c006b40).stop 2024-01-21T11:16:16.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 msgr2=0x7f4b4c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:16.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c106720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4b40013510 tx=0x7f4b4000b040).stop 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 shutdown_connections 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4b3402ff60 0x7f4b34032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 --2- 172.21.15.178:0/107605804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b4c0f5410 0x7f4b4c106720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 >> 172.21.15.178:0/107605804 conn(0x7f4b4c0f0b70 msgr2=0x7f4b4c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 shutdown_connections 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:16.546+0000 7f4b5096b700 1 -- 172.21.15.178:0/107605804 wait complete. 2024-01-21T11:16:16.548 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:16.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:15.665657+0000 mgr.smithi089.ufybqw (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:16.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:16.546518+0000 mon.smithi089 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.178:0/107605804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:17.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:17.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:19.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:17.665901+0000 mgr.smithi089.ufybqw (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 -- 172.21.15.178:0/2514946141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 msgr2=0x7f82b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 --2- 172.21.15.178:0/2514946141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b80f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f82b0009960 tx=0x7f82b0004d10).stop 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 -- 172.21.15.178:0/2514946141 shutdown_connections 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 --2- 172.21.15.178:0/2514946141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b80f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 -- 172.21.15.178:0/2514946141 >> 172.21.15.178:0/2514946141 conn(0x7f82b80f0b70 msgr2=0x7f82b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 -- 172.21.15.178:0/2514946141 shutdown_connections 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.196+0000 7f82bfbf9700 1 -- 172.21.15.178:0/2514946141 wait complete. 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bfbf9700 1 Processor -- start 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bfbf9700 1 -- start start 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bfbf9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b8106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bfbf9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b8106c00 con 0x7f82b80f5410 2024-01-21T11:16:19.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bd995700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b8106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bd995700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b8106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55066/0 (socket says 172.21.15.178:55066) 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bd995700 1 -- 172.21.15.178:0/1802556193 learned_addr learned my addr 172.21.15.178:0/1802556193 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.197+0000 7f82bd995700 1 -- 172.21.15.178:0/1802556193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b0009640 con 0x7f82b80f5410 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.198+0000 7f82bd995700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b8106710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f82b0004b70 tx=0x7f82b0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.198+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82b0018070 con 0x7f82b80f5410 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.198+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b8106e00 con 0x7f82b80f5410 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.198+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82b000b870 con 0x7f82b80f5410 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.198+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82b001c560 con 0x7f82b80f5410 2024-01-21T11:16:19.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.198+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b8107290 con 0x7f82b80f5410 2024-01-21T11:16:19.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.199+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82b001c6c0 con 0x7f82b80f5410 2024-01-21T11:16:19.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.199+0000 7f82ae7fc700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f82a402ff60 0x7f82a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:19.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.199+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82b00412e0 con 0x7f82b80f5410 2024-01-21T11:16:19.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.199+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b8045ad0 con 0x7f82b80f5410 2024-01-21T11:16:19.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.199+0000 7f82bd194700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f82a402ff60 0x7f82a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:19.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.199+0000 7f82bd194700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f82a402ff60 0x7f82a4032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f82a80096a0 tx=0x7f82a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:19.204 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.203+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82b00127f0 con 0x7f82b80f5410 2024-01-21T11:16:19.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.409+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82b8107fb0 con 0x7f82b80f5410 2024-01-21T11:16:19.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.410+0000 7f82ae7fc700 1 -- 172.21.15.178:0/1802556193 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82b0019230 con 0x7f82b80f5410 2024-01-21T11:16:19.411 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:19.411 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:19.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f82a402ff60 msgr2=0x7f82a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:19.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f82a402ff60 0x7f82a4032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f82a80096a0 tx=0x7f82a8006b40).stop 2024-01-21T11:16:19.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 msgr2=0x7f82b8106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b8106710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f82b0004b70 tx=0x7f82b0004d10).stop 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 shutdown_connections 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f82a402ff60 0x7f82a4032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 --2- 172.21.15.178:0/1802556193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82b80f5410 0x7f82b8106710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 >> 172.21.15.178:0/1802556193 conn(0x7f82b80f0b70 msgr2=0x7f82b805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 shutdown_connections 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:19.411+0000 7f82bfbf9700 1 -- 172.21.15.178:0/1802556193 wait complete. 2024-01-21T11:16:19.413 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:20.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:19 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:19.411243+0000 mon.smithi089 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.178:0/1802556193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:20.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:20.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:21.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:19.666211+0000 mgr.smithi089.ufybqw (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:21.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 -- 172.21.15.178:0/4175189118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 msgr2=0x7fb3640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 --2- 172.21.15.178:0/4175189118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640f55c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fb358009fb0 tx=0x7fb358009b30).stop 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 -- 172.21.15.178:0/4175189118 shutdown_connections 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 --2- 172.21.15.178:0/4175189118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640f55c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 -- 172.21.15.178:0/4175189118 >> 172.21.15.178:0/4175189118 conn(0x7fb3640f09a0 msgr2=0x7fb3640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 -- 172.21.15.178:0/4175189118 shutdown_connections 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.947+0000 7fb3695f8700 1 -- 172.21.15.178:0/4175189118 wait complete. 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb3695f8700 1 Processor -- start 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb3695f8700 1 -- start start 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb3695f8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb3695f8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640fb5a0 con 0x7fb3640f51a0 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb363fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb363fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55082/0 (socket says 172.21.15.178:55082) 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.948+0000 7fb363fff700 1 -- 172.21.15.178:0/3622255217 learned_addr learned my addr 172.21.15.178:0/3622255217 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb363fff700 1 -- 172.21.15.178:0/3622255217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358005040 con 0x7fb3640f51a0 2024-01-21T11:16:21.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb363fff700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640fd120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb358013510 tx=0x7fb35800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:21.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358009320 con 0x7fb3640f51a0 2024-01-21T11:16:21.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3640fb740 con 0x7fb3640f51a0 2024-01-21T11:16:21.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb358013f00 con 0x7fb3640f51a0 2024-01-21T11:16:21.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358014dc0 con 0x7fb3640f51a0 2024-01-21T11:16:21.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.949+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3640fbbd0 con 0x7fb3640f51a0 2024-01-21T11:16:21.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.950+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3580095f0 con 0x7fb3640f51a0 2024-01-21T11:16:21.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.950+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb364059100 con 0x7fb3640f51a0 2024-01-21T11:16:21.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.950+0000 7fb360ff9700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb34c02ff60 0x7fb34c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:21.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.950+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb358041040 con 0x7fb3640f51a0 2024-01-21T11:16:21.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.950+0000 7fb3637fe700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb34c02ff60 0x7fb34c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:21.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.951+0000 7fb3637fe700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb34c02ff60 0x7fb34c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:21.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:21.955+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb358014420 con 0x7fb3640f51a0 2024-01-21T11:16:22.163 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.163+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3640fabe0 con 0x7fb3640f51a0 2024-01-21T11:16:22.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.163+0000 7fb360ff9700 1 -- 172.21.15.178:0/3622255217 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb358010230 con 0x7fb3640f51a0 2024-01-21T11:16:22.165 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:22.165 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb34c02ff60 msgr2=0x7fb34c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb34c02ff60 0x7fb34c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).stop 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 msgr2=0x7fb3640fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640fd120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb358013510 tx=0x7fb35800b040).stop 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 shutdown_connections 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb34c02ff60 0x7fb34c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 --2- 172.21.15.178:0/3622255217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3640f51a0 0x7fb3640fd120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:22.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 >> 172.21.15.178:0/3622255217 conn(0x7fb3640f09a0 msgr2=0x7fb36405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:22.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 shutdown_connections 2024-01-21T11:16:22.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:22.165+0000 7fb3695f8700 1 -- 172.21.15.178:0/3622255217 wait complete. 2024-01-21T11:16:22.167 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:21.666551+0000 mgr.smithi089.ufybqw (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:22.165163+0000 mon.smithi089 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.178:0/3622255217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:23.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:23.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 -- 172.21.15.178:0/1960258192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 msgr2=0x7fa5440f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 --2- 172.21.15.178:0/1960258192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa5440f6e00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fa538009fb0 tx=0x7fa538009b30).stop 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 -- 172.21.15.178:0/1960258192 shutdown_connections 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 --2- 172.21.15.178:0/1960258192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa5440f6e00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 -- 172.21.15.178:0/1960258192 >> 172.21.15.178:0/1960258192 conn(0x7fa5440f02e0 msgr2=0x7fa5440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 -- 172.21.15.178:0/1960258192 shutdown_connections 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 -- 172.21.15.178:0/1960258192 wait complete. 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.727+0000 7fa548a18700 1 Processor -- start 2024-01-21T11:16:24.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa548a18700 1 -- start start 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa548a18700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa544106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa548a18700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544106bf0 con 0x7fa5440f69e0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa54259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa544106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa54259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa544106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53164/0 (socket says 172.21.15.178:53164) 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa54259c700 1 -- 172.21.15.178:0/2567445738 learned_addr learned my addr 172.21.15.178:0/2567445738 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa54259c700 1 -- 172.21.15.178:0/2567445738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538005040 con 0x7fa5440f69e0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa54259c700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa544106700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fa538013510 tx=0x7fa53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.728+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa538009320 con 0x7fa5440f69e0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa544106df0 con 0x7fa5440f69e0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa538013f00 con 0x7fa5440f69e0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa538014d90 con 0x7fa5440f69e0 2024-01-21T11:16:24.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa544107280 con 0x7fa5440f69e0 2024-01-21T11:16:24.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5380095f0 con 0x7fa5440f69e0 2024-01-21T11:16:24.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa544045ad0 con 0x7fa5440f69e0 2024-01-21T11:16:24.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.729+0000 7fa532ffd700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa52c02ff60 0x7fa52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:24.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.730+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa538041080 con 0x7fa5440f69e0 2024-01-21T11:16:24.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.730+0000 7fa541d9b700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa52c02ff60 0x7fa52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:24.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.730+0000 7fa541d9b700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa52c02ff60 0x7fa52c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa5340096a0 tx=0x7fa534006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:24.735 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.734+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa538014420 con 0x7fa5440f69e0 2024-01-21T11:16:24.941 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.940+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5441080d0 con 0x7fa5440f69e0 2024-01-21T11:16:24.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.942+0000 7fa532ffd700 1 -- 172.21.15.178:0/2567445738 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa538010230 con 0x7fa5440f69e0 2024-01-21T11:16:24.942 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:24.942 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:24.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa52c02ff60 msgr2=0x7fa52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa52c02ff60 0x7fa52c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa5340096a0 tx=0x7fa534006b40).stop 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 msgr2=0x7fa544106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa544106700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fa538013510 tx=0x7fa53800b040).stop 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 shutdown_connections 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa52c02ff60 0x7fa52c032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 --2- 172.21.15.178:0/2567445738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5440f69e0 0x7fa544106700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 >> 172.21.15.178:0/2567445738 conn(0x7fa5440f02e0 msgr2=0x7fa5440f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 shutdown_connections 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:24.943+0000 7fa548a18700 1 -- 172.21.15.178:0/2567445738 wait complete. 2024-01-21T11:16:24.944 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:25.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:23.667122+0000 mgr.smithi089.ufybqw (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:26.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:25 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:24.942569+0000 mon.smithi089 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.178:0/2567445738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:26.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:25 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:25.667560+0000 mgr.smithi089.ufybqw (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:26.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:26.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:27.533 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3498119051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 msgr2=0x7fc4d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:27.533 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 --2- 172.21.15.178:0/3498119051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d00f55c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fc4c4009fb0 tx=0x7fc4c4009b30).stop 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3498119051 shutdown_connections 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 --2- 172.21.15.178:0/3498119051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d00f55c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3498119051 >> 172.21.15.178:0/3498119051 conn(0x7fc4d00f09a0 msgr2=0x7fc4d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3498119051 shutdown_connections 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3498119051 wait complete. 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 Processor -- start 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.531+0000 7fc4d5f80700 1 -- start start 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d5f80700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d01064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d5f80700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d01069b0 con 0x7fc4d00f51a0 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d4f7e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d01064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d4f7e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d01064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53188/0 (socket says 172.21.15.178:53188) 2024-01-21T11:16:27.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d4f7e700 1 -- 172.21.15.178:0/3403799345 learned_addr learned my addr 172.21.15.178:0/3403799345 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d4f7e700 1 -- 172.21.15.178:0/3403799345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4005040 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.532+0000 7fc4d4f7e700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d01064c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc4c4013510 tx=0x7fc4c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.533+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c4009320 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.533+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4c4013f00 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.533+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d0106bb0 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.533+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c4014dc0 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.533+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d0107040 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.533+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4c4014420 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.534+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d0045ad0 con 0x7fc4d00f51a0 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.534+0000 7fc4cd7fa700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc4c002ffb0 0x7fc4c0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:27.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.534+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4c4041110 con 0x7fc4d00f51a0 2024-01-21T11:16:27.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.534+0000 7fc4cffff700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc4c002ffb0 0x7fc4c0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:27.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.534+0000 7fc4cffff700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc4c002ffb0 0x7fc4c0032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc4bc0096a0 tx=0x7fc4bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:27.539 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.539+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4c40095f0 con 0x7fc4d00f51a0 2024-01-21T11:16:27.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.742+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4d0107ef0 con 0x7fc4d00f51a0 2024-01-21T11:16:27.743 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.742+0000 7fc4cd7fa700 1 -- 172.21.15.178:0/3403799345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4c401cb10 con 0x7fc4d00f51a0 2024-01-21T11:16:27.743 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:27.743 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:27.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc4c002ffb0 msgr2=0x7fc4c0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:27.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc4c002ffb0 0x7fc4c0032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc4bc0096a0 tx=0x7fc4bc006b40).stop 2024-01-21T11:16:27.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 msgr2=0x7fc4d01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:27.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d01064c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc4c4013510 tx=0x7fc4c400b040).stop 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 shutdown_connections 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc4c002ffb0 0x7fc4c0032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 --2- 172.21.15.178:0/3403799345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4d00f51a0 0x7fc4d01064c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 >> 172.21.15.178:0/3403799345 conn(0x7fc4d00f09a0 msgr2=0x7fc4d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 shutdown_connections 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:27.744+0000 7fc4d5f80700 1 -- 172.21.15.178:0/3403799345 wait complete. 2024-01-21T11:16:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:28.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:27.667823+0000 mgr.smithi089.ufybqw (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:28.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:27.744112+0000 mon.smithi089 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.178:0/3403799345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:29.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:29.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:30.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- 172.21.15.178:0/2484168458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b00f6510 msgr2=0x7f74b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 --2- 172.21.15.178:0/2484168458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b00f6510 0x7f74b00f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f74a0009fb0 tx=0x7f74a0009b30).stop 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- 172.21.15.178:0/2484168458 shutdown_connections 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 --2- 172.21.15.178:0/2484168458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b00f6510 0x7f74b00f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- 172.21.15.178:0/2484168458 >> 172.21.15.178:0/2484168458 conn(0x7f74b00f1c70 msgr2=0x7f74b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- 172.21.15.178:0/2484168458 shutdown_connections 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- 172.21.15.178:0/2484168458 wait complete. 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 Processor -- start 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- start start 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 0x7f74b0106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:30.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74b5a4a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0107100 con 0x7f74b01067f0 2024-01-21T11:16:30.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 0x7f74b0106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:30.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 0x7f74b0106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:53220/0 (socket says 172.21.15.178:53220) 2024-01-21T11:16:30.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74aeffd700 1 -- 172.21.15.178:0/1679697984 learned_addr learned my addr 172.21.15.178:0/1679697984 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:30.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74aeffd700 1 -- 172.21.15.178:0/1679697984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0005040 con 0x7f74b01067f0 2024-01-21T11:16:30.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.352+0000 7f74aeffd700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 0x7f74b0106c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f74a0009f80 tx=0x7f74a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:30.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.353+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74a00136c0 con 0x7f74b01067f0 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.353+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74a00095f0 con 0x7f74b01067f0 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.353+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b0107300 con 0x7f74b01067f0 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.353+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74a0014420 con 0x7f74b01067f0 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.353+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b0107e50 con 0x7f74b01067f0 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.353+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74a0025060 con 0x7f74b01067f0 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.354+0000 7f74977fe700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f749802ff60 0x7f7498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:30.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.354+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74a0040dd0 con 0x7f74b01067f0 2024-01-21T11:16:30.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.354+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b0059100 con 0x7f74b01067f0 2024-01-21T11:16:30.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.354+0000 7f74ae7fc700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f749802ff60 0x7f7498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:30.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.354+0000 7f74ae7fc700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f749802ff60 0x7f7498032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f74a40096a0 tx=0x7f74a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:30.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.359+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74a0028440 con 0x7f74b01067f0 2024-01-21T11:16:30.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.563+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74b0108550 con 0x7f74b01067f0 2024-01-21T11:16:30.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.563+0000 7f74977fe700 1 -- 172.21.15.178:0/1679697984 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74a001d3a0 con 0x7f74b01067f0 2024-01-21T11:16:30.564 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:30.564 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:30.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.564+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f749802ff60 msgr2=0x7f7498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:30.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.564+0000 7f74b5a4a700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f749802ff60 0x7f7498032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f74a40096a0 tx=0x7f74a4006b40).stop 2024-01-21T11:16:30.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.564+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 msgr2=0x7f74b0106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.564+0000 7f74b5a4a700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 0x7f74b0106c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f74a0009f80 tx=0x7f74a0009b30).stop 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.565+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 shutdown_connections 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.565+0000 7f74b5a4a700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f749802ff60 0x7f7498032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.565+0000 7f74b5a4a700 1 --2- 172.21.15.178:0/1679697984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b01067f0 0x7f74b0106c10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.565+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 >> 172.21.15.178:0/1679697984 conn(0x7f74b00f1c70 msgr2=0x7f74b00f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.565+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 shutdown_connections 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:30.565+0000 7f74b5a4a700 1 -- 172.21.15.178:0/1679697984 wait complete. 2024-01-21T11:16:30.566 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:30.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:29.668133+0000 mgr.smithi089.ufybqw (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:30.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:30.564855+0000 mon.smithi089 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.178:0/1679697984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:31.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:31.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:32.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:31.668463+0000 mgr.smithi089.ufybqw (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.170+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2126309329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 msgr2=0x7fe0d00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.170+0000 7fe0d6d1a700 1 --2- 172.21.15.178:0/2126309329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d00f9150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fe0c4009fb0 tx=0x7fe0c4009b30).stop 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.170+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2126309329 shutdown_connections 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.170+0000 7fe0d6d1a700 1 --2- 172.21.15.178:0/2126309329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d00f9150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.170+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2126309329 >> 172.21.15.178:0/2126309329 conn(0x7fe0d00f0b70 msgr2=0x7fe0d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.170+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2126309329 shutdown_connections 2024-01-21T11:16:33.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.171+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2126309329 wait complete. 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.171+0000 7fe0d6d1a700 1 Processor -- start 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.171+0000 7fe0d6d1a700 1 -- start start 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.171+0000 7fe0d6d1a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.171+0000 7fe0d6d1a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0106bd0 con 0x7fe0d0060220 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d4ab6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d4ab6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:45160/0 (socket says 172.21.15.178:45160) 2024-01-21T11:16:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d4ab6700 1 -- 172.21.15.178:0/2821130726 learned_addr learned my addr 172.21.15.178:0/2821130726 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:33.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d4ab6700 1 -- 172.21.15.178:0/2821130726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4005040 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d4ab6700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d01066e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe0c4013510 tx=0x7fe0c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c4009320 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d0106dd0 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0c4013f00 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c4014dc0 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.172+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d0107260 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.173+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0c4014420 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.173+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0045ad0 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.173+0000 7fe0cd7fa700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe0c002ff60 0x7fe0c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.173+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0c40410e0 con 0x7fe0d0060220 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.173+0000 7fe0cffff700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe0c002ff60 0x7fe0c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:33.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.174+0000 7fe0cffff700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe0c002ff60 0x7fe0c0032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe0bc0096a0 tx=0x7fe0bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:33.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.178+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0c40095f0 con 0x7fe0d0060220 2024-01-21T11:16:33.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.384+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0d01080b0 con 0x7fe0d0060220 2024-01-21T11:16:33.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.385+0000 7fe0cd7fa700 1 -- 172.21.15.178:0/2821130726 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0c401cb10 con 0x7fe0d0060220 2024-01-21T11:16:33.384 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:33.385 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe0c002ff60 msgr2=0x7fe0c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe0c002ff60 0x7fe0c0032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe0bc0096a0 tx=0x7fe0bc006b40).stop 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 msgr2=0x7fe0d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d01066e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe0c4013510 tx=0x7fe0c400b040).stop 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 shutdown_connections 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe0c002ff60 0x7fe0c0032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 --2- 172.21.15.178:0/2821130726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0d0060220 0x7fe0d01066e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:33.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 >> 172.21.15.178:0/2821130726 conn(0x7fe0d00f0b70 msgr2=0x7fe0d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:33.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 shutdown_connections 2024-01-21T11:16:33.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:33.386+0000 7fe0d6d1a700 1 -- 172.21.15.178:0/2821130726 wait complete. 2024-01-21T11:16:33.387 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:33.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:33 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:33.385338+0000 mon.smithi089 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.178:0/2821130726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:34.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:34.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:34.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:33.668844+0000 mgr.smithi089.ufybqw (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:36.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/1372606003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 msgr2=0x7fe5d40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:36.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 --2- 172.21.15.178:0/1372606003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d40f4b60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe5d0009fb0 tx=0x7fe5d0009b30).stop 2024-01-21T11:16:36.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/1372606003 shutdown_connections 2024-01-21T11:16:36.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 --2- 172.21.15.178:0/1372606003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d40f4b60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:36.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/1372606003 >> 172.21.15.178:0/1372606003 conn(0x7fe5d40f0260 msgr2=0x7fe5d40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/1372606003 shutdown_connections 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/1372606003 wait complete. 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 Processor -- start 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- start start 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.004+0000 7fe5dc5d6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d4106bc0 con 0x7fe5d40f4740 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5da372700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5da372700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:45166/0 (socket says 172.21.15.178:45166) 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5da372700 1 -- 172.21.15.178:0/3361192521 learned_addr learned my addr 172.21.15.178:0/3361192521 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5da372700 1 -- 172.21.15.178:0/3361192521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d0005040 con 0x7fe5d40f4740 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5da372700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d41066d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe5d0013510 tx=0x7fe5d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d0009320 con 0x7fe5d40f4740 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d4106dc0 con 0x7fe5d40f4740 2024-01-21T11:16:36.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d4107250 con 0x7fe5d40f4740 2024-01-21T11:16:36.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5d0013f00 con 0x7fe5d40f4740 2024-01-21T11:16:36.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.005+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d0014cc0 con 0x7fe5d40f4740 2024-01-21T11:16:36.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.006+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc004fa0 con 0x7fe5d40f4740 2024-01-21T11:16:36.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.006+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5d0014420 con 0x7fe5d40f4740 2024-01-21T11:16:36.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.006+0000 7fe5caffd700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe5c002ff60 0x7fe5c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:36.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.006+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5d0040f50 con 0x7fe5d40f4740 2024-01-21T11:16:36.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.007+0000 7fe5d9b71700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe5c002ff60 0x7fe5c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:36.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.007+0000 7fe5d9b71700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe5c002ff60 0x7fe5c0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe5c40096a0 tx=0x7fe5c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:36.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.011+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5d006d2b0 con 0x7fe5d40f4740 2024-01-21T11:16:36.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.216+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5bc005670 con 0x7fe5d40f4740 2024-01-21T11:16:36.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.217+0000 7fe5caffd700 1 -- 172.21.15.178:0/3361192521 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe5d001db70 con 0x7fe5d40f4740 2024-01-21T11:16:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:36.217 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:36.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe5c002ff60 msgr2=0x7fe5c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:36.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe5c002ff60 0x7fe5c0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe5c40096a0 tx=0x7fe5c4006b40).stop 2024-01-21T11:16:36.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 msgr2=0x7fe5d41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:36.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d41066d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe5d0013510 tx=0x7fe5d000b040).stop 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 shutdown_connections 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe5c002ff60 0x7fe5c0032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 --2- 172.21.15.178:0/3361192521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d40f4740 0x7fe5d41066d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 >> 172.21.15.178:0/3361192521 conn(0x7fe5d40f0260 msgr2=0x7fe5d40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 shutdown_connections 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:36.218+0000 7fe5dc5d6700 1 -- 172.21.15.178:0/3361192521 wait complete. 2024-01-21T11:16:36.219 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:36.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:35.669199+0000 mgr.smithi089.ufybqw (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:36.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:36.217470+0000 mon.smithi089 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.178:0/3361192521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:37.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:37.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:38.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 -- 172.21.15.178:0/610965108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 msgr2=0x7fcfa00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 --2- 172.21.15.178:0/610965108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa00f66c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fcf94009fb0 tx=0x7fcf94009b30).stop 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 -- 172.21.15.178:0/610965108 shutdown_connections 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 --2- 172.21.15.178:0/610965108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa00f66c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 -- 172.21.15.178:0/610965108 >> 172.21.15.178:0/610965108 conn(0x7fcfa00f1aa0 msgr2=0x7fcfa00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 -- 172.21.15.178:0/610965108 shutdown_connections 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 -- 172.21.15.178:0/610965108 wait complete. 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 Processor -- start 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.735+0000 7fcf9f59e700 1 -- start start 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9f59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9f59e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa01069a0 con 0x7fcfa00f62a0 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:45180/0 (socket says 172.21.15.178:45180) 2024-01-21T11:16:38.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9e59c700 1 -- 172.21.15.178:0/1092663927 learned_addr learned my addr 172.21.15.178:0/1092663927 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9e59c700 1 -- 172.21.15.178:0/1092663927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94005040 con 0x7fcfa00f62a0 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9e59c700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa01064b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fcf94009830 tx=0x7fcf9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf94009320 con 0x7fcfa00f62a0 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa0106ba0 con 0x7fcfa00f62a0 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.736+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa0107030 con 0x7fcfa00f62a0 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.737+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf940136c0 con 0x7fcfa00f62a0 2024-01-21T11:16:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.737+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9401ee90 con 0x7fcfa00f62a0 2024-01-21T11:16:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.737+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf94014420 con 0x7fcfa00f62a0 2024-01-21T11:16:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.737+0000 7fcf8effd700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcf8802ff10 0x7fcf880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.737+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf94040f90 con 0x7fcfa00f62a0 2024-01-21T11:16:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.737+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa0045ad0 con 0x7fcfa00f62a0 2024-01-21T11:16:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.738+0000 7fcf9dd9b700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcf8802ff10 0x7fcf880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.738+0000 7fcf9dd9b700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcf8802ff10 0x7fcf880323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fcf900096a0 tx=0x7fcf90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.742+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf94027770 con 0x7fcfa00f62a0 2024-01-21T11:16:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.949+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfa00fbce0 con 0x7fcfa00f62a0 2024-01-21T11:16:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.950+0000 7fcf8effd700 1 -- 172.21.15.178:0/1092663927 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf9401a230 con 0x7fcfa00f62a0 2024-01-21T11:16:38.950 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:38.950 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:38.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcf8802ff10 msgr2=0x7fcf880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:38.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcf8802ff10 0x7fcf880323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fcf900096a0 tx=0x7fcf90006b40).stop 2024-01-21T11:16:38.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 msgr2=0x7fcfa01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:38.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa01064b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fcf94009830 tx=0x7fcf9400b040).stop 2024-01-21T11:16:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 shutdown_connections 2024-01-21T11:16:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcf8802ff10 0x7fcf880323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 --2- 172.21.15.178:0/1092663927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00f62a0 0x7fcfa01064b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 >> 172.21.15.178:0/1092663927 conn(0x7fcfa00f1aa0 msgr2=0x7fcfa00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 shutdown_connections 2024-01-21T11:16:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:38.951+0000 7fcf9f59e700 1 -- 172.21.15.178:0/1092663927 wait complete. 2024-01-21T11:16:38.953 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:39.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:37.669566+0000 mgr.smithi089.ufybqw (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:40.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:38.950335+0000 mon.smithi089 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.178:0/1092663927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:40.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:40.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:41.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:39.669829+0000 mgr.smithi089.ufybqw (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:41.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.484+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3649998585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 msgr2=0x7fdab40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.484+0000 7fdabb0ab700 1 --2- 172.21.15.178:0/3649998585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab40f55c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fdaa4009fb0 tx=0x7fdaa4009b30).stop 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.484+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3649998585 shutdown_connections 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.484+0000 7fdabb0ab700 1 --2- 172.21.15.178:0/3649998585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab40f55c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.484+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3649998585 >> 172.21.15.178:0/3649998585 conn(0x7fdab40f09a0 msgr2=0x7fdab40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3649998585 shutdown_connections 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3649998585 wait complete. 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdabb0ab700 1 Processor -- start 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdabb0ab700 1 -- start start 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdabb0ab700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdabb0ab700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab4106990 con 0x7fdab40f51a0 2024-01-21T11:16:41.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdaba0a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdaba0a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:45186/0 (socket says 172.21.15.178:45186) 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.485+0000 7fdaba0a9700 1 -- 172.21.15.178:0/3213195053 learned_addr learned my addr 172.21.15.178:0/3213195053 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdaba0a9700 1 -- 172.21.15.178:0/3213195053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4005040 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdaba0a9700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab41064a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fdaa4013510 tx=0x7fdaa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa4009320 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab4106b90 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaa4013f00 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa4014dc0 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.486+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab4107020 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.487+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaa40095f0 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.487+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab4045ad0 con 0x7fdab40f51a0 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.487+0000 7fdaaaffd700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdaa002ff60 0x7fdaa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.487+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaa4041000 con 0x7fdab40f51a0 2024-01-21T11:16:41.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.487+0000 7fdab98a8700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdaa002ff60 0x7fdaa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:41.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.488+0000 7fdab98a8700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdaa002ff60 0x7fdaa0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdab00096a0 tx=0x7fdab0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:41.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.492+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaa4014420 con 0x7fdab40f51a0 2024-01-21T11:16:41.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.697+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdab4107ed0 con 0x7fdab40f51a0 2024-01-21T11:16:41.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.698+0000 7fdaaaffd700 1 -- 172.21.15.178:0/3213195053 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaa4010230 con 0x7fdab40f51a0 2024-01-21T11:16:41.698 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:41.698 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdaa002ff60 msgr2=0x7fdaa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdaa002ff60 0x7fdaa0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdab00096a0 tx=0x7fdab0006b40).stop 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 msgr2=0x7fdab41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab41064a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fdaa4013510 tx=0x7fdaa400b040).stop 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 shutdown_connections 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdaa002ff60 0x7fdaa0032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 --2- 172.21.15.178:0/3213195053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab40f51a0 0x7fdab41064a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 >> 172.21.15.178:0/3213195053 conn(0x7fdab40f09a0 msgr2=0x7fdab405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 shutdown_connections 2024-01-21T11:16:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:41.700+0000 7fdabb0ab700 1 -- 172.21.15.178:0/3213195053 wait complete. 2024-01-21T11:16:41.701 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:43.108 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:43.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:43.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:41.670122+0000 mgr.smithi089.ufybqw (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:43.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:41.698788+0000 mon.smithi089 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.178:0/3213195053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:44.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- 172.21.15.178:0/1515609305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 msgr2=0x7fa2c00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:44.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 --2- 172.21.15.178:0/1515609305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c00f8f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fa2bc009fb0 tx=0x7fa2bc009b30).stop 2024-01-21T11:16:44.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- 172.21.15.178:0/1515609305 shutdown_connections 2024-01-21T11:16:44.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 --2- 172.21.15.178:0/1515609305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c00f8f20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:44.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- 172.21.15.178:0/1515609305 >> 172.21.15.178:0/1515609305 conn(0x7fa2c00f09a0 msgr2=0x7fa2c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- 172.21.15.178:0/1515609305 shutdown_connections 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- 172.21.15.178:0/1515609305 wait complete. 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 Processor -- start 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- start start 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c0106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.265+0000 7fa2c676d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0106a10 con 0x7fa2c00601e0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c576b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c0106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c576b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c0106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41636/0 (socket says 172.21.15.178:41636) 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c576b700 1 -- 172.21.15.178:0/3069869173 learned_addr learned my addr 172.21.15.178:0/3069869173 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c576b700 1 -- 172.21.15.178:0/3069869173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc005040 con 0x7fa2c00601e0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c576b700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c0106520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fa2bc014000 tx=0x7fa2bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2bc0095f0 con 0x7fa2c00601e0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c0106c10 con 0x7fa2c00601e0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2bc007e70 con 0x7fa2c00601e0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2bc014ea0 con 0x7fa2c00601e0 2024-01-21T11:16:44.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.266+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c01070a0 con 0x7fa2c00601e0 2024-01-21T11:16:44.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.267+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2bc025070 con 0x7fa2c00601e0 2024-01-21T11:16:44.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.267+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0045ad0 con 0x7fa2c00601e0 2024-01-21T11:16:44.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.267+0000 7fa2b5ffb700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa2ac02ff60 0x7fa2ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:44.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.267+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2bc0409c0 con 0x7fa2c00601e0 2024-01-21T11:16:44.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.268+0000 7fa2c4f6a700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa2ac02ff60 0x7fa2ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:44.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.268+0000 7fa2c4f6a700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa2ac02ff60 0x7fa2ac032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fa2b00096a0 tx=0x7fa2b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:44.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.272+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2bc0079c0 con 0x7fa2c00601e0 2024-01-21T11:16:44.479 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.479+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2c0107ef0 con 0x7fa2c00601e0 2024-01-21T11:16:44.479 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.480+0000 7fa2b5ffb700 1 -- 172.21.15.178:0/3069869173 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2bc00f300 con 0x7fa2c00601e0 2024-01-21T11:16:44.480 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:44.480 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa2ac02ff60 msgr2=0x7fa2ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa2ac02ff60 0x7fa2ac032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fa2b00096a0 tx=0x7fa2b0006b40).stop 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 msgr2=0x7fa2c0106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c0106520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fa2bc014000 tx=0x7fa2bc00b040).stop 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 shutdown_connections 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa2ac02ff60 0x7fa2ac032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:44.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 --2- 172.21.15.178:0/3069869173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2c00601e0 0x7fa2c0106520 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:44.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 >> 172.21.15.178:0/3069869173 conn(0x7fa2c00f09a0 msgr2=0x7fa2c00f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:44.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 shutdown_connections 2024-01-21T11:16:44.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:44.481+0000 7fa2c676d700 1 -- 172.21.15.178:0/3069869173 wait complete. 2024-01-21T11:16:44.482 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:45.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:43.670366+0000 mgr.smithi089.ufybqw (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:45.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:44.480201+0000 mon.smithi089 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.178:0/3069869173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:45.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:45.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:47.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.101+0000 7f73a335e700 1 -- 172.21.15.178:0/1766795538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 msgr2=0x7f739c0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:47.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.101+0000 7f73a335e700 1 --2- 172.21.15.178:0/1766795538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c0f8fc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f738c009fb0 tx=0x7f738c009b30).stop 2024-01-21T11:16:47.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.101+0000 7f73a335e700 1 -- 172.21.15.178:0/1766795538 shutdown_connections 2024-01-21T11:16:47.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.101+0000 7f73a335e700 1 --2- 172.21.15.178:0/1766795538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c0f8fc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.101+0000 7f73a335e700 1 -- 172.21.15.178:0/1766795538 >> 172.21.15.178:0/1766795538 conn(0x7f739c0f09a0 msgr2=0x7f739c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a335e700 1 -- 172.21.15.178:0/1766795538 shutdown_connections 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a335e700 1 -- 172.21.15.178:0/1766795538 wait complete. 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a335e700 1 Processor -- start 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a335e700 1 -- start start 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a335e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a335e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c106a10 con 0x7f739c060140 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.102+0000 7f73a235c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f73a235c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41648/0 (socket says 172.21.15.178:41648) 2024-01-21T11:16:47.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f73a235c700 1 -- 172.21.15.178:0/2024624336 learned_addr learned my addr 172.21.15.178:0/2024624336 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f73a235c700 1 -- 172.21.15.178:0/2024624336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c005040 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f73a235c700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c106520 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f738c014000 tx=0x7f738c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f738c0095f0 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c106c10 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f738c007e70 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f738c014ea0 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.103+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c1070a0 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.104+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f738c025070 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.104+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739c045ad0 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.104+0000 7f7392ffd700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f738802ffb0 0x7f7388032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.104+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f738c0409e0 con 0x7f739c060140 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.105+0000 7f73a1b5b700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f738802ffb0 0x7f7388032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:47.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.105+0000 7f73a1b5b700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f738802ffb0 0x7f7388032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f73980096a0 tx=0x7f7398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:47.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.109+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f738c00f4e0 con 0x7f739c060140 2024-01-21T11:16:47.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:45.670697+0000 mgr.smithi089.ufybqw (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:47.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.316+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f739c107f50 con 0x7f739c060140 2024-01-21T11:16:47.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.316+0000 7f7392ffd700 1 -- 172.21.15.178:0/2024624336 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f738c02d090 con 0x7f739c060140 2024-01-21T11:16:47.316 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:47.317 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f738802ffb0 msgr2=0x7f7388032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f738802ffb0 0x7f7388032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f73980096a0 tx=0x7f7398006b40).stop 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 msgr2=0x7f739c106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c106520 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f738c014000 tx=0x7f738c00b040).stop 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 shutdown_connections 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f738802ffb0 0x7f7388032470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 --2- 172.21.15.178:0/2024624336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f739c060140 0x7f739c106520 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:47.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 >> 172.21.15.178:0/2024624336 conn(0x7f739c0f09a0 msgr2=0x7f739c0f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:47.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 shutdown_connections 2024-01-21T11:16:47.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:47.318+0000 7f73a335e700 1 -- 172.21.15.178:0/2024624336 wait complete. 2024-01-21T11:16:47.319 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:48.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:47.317162+0000 mon.smithi089 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.178:0/2024624336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:48.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:48.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:49.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:47.671087+0000 mgr.smithi089.ufybqw (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 -- 172.21.15.178:0/1863647863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 msgr2=0x7fc0440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 --2- 172.21.15.178:0/1863647863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0440f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc034009fb0 tx=0x7fc034009b30).stop 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 -- 172.21.15.178:0/1863647863 shutdown_connections 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 --2- 172.21.15.178:0/1863647863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0440f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 -- 172.21.15.178:0/1863647863 >> 172.21.15.178:0/1863647863 conn(0x7fc0440f0b70 msgr2=0x7fc0440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 -- 172.21.15.178:0/1863647863 shutdown_connections 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.952+0000 7fc04a254700 1 -- 172.21.15.178:0/1863647863 wait complete. 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc04a254700 1 Processor -- start 2024-01-21T11:16:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc04a254700 1 -- start start 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc04a254700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0441066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc04a254700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc044106ba0 con 0x7fc0440f5410 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc0437fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0441066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc0437fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0441066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41670/0 (socket says 172.21.15.178:41670) 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.953+0000 7fc0437fe700 1 -- 172.21.15.178:0/958997934 learned_addr learned my addr 172.21.15.178:0/958997934 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc0437fe700 1 -- 172.21.15.178:0/958997934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc034005040 con 0x7fc0440f5410 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc0437fe700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0441066b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc034014000 tx=0x7fc03400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0340095f0 con 0x7fc0440f5410 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc044106da0 con 0x7fc0440f5410 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc034007e70 con 0x7fc0440f5410 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc034014ea0 con 0x7fc0440f5410 2024-01-21T11:16:49.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.954+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc044107230 con 0x7fc0440f5410 2024-01-21T11:16:49.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.955+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc034025070 con 0x7fc0440f5410 2024-01-21T11:16:49.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.955+0000 7fc040ff9700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc03002ff60 0x7fc030032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:49.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.955+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc044045ad0 con 0x7fc0440f5410 2024-01-21T11:16:49.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.955+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0340409f0 con 0x7fc0440f5410 2024-01-21T11:16:49.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.955+0000 7fc03affd700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc03002ff60 0x7fc030032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:49.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.956+0000 7fc03affd700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc03002ff60 0x7fc030032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc02c0096a0 tx=0x7fc02c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:49.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:49.960+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc03400f4e0 con 0x7fc0440f5410 2024-01-21T11:16:50.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.165+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0441080e0 con 0x7fc0440f5410 2024-01-21T11:16:50.165 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.165+0000 7fc040ff9700 1 -- 172.21.15.178:0/958997934 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc03402d090 con 0x7fc0440f5410 2024-01-21T11:16:50.165 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:50.166 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc03002ff60 msgr2=0x7fc030032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc03002ff60 0x7fc030032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc02c0096a0 tx=0x7fc02c006b40).stop 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 msgr2=0x7fc0441066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0441066b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc034014000 tx=0x7fc03400b040).stop 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 shutdown_connections 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc03002ff60 0x7fc030032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 --2- 172.21.15.178:0/958997934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0440f5410 0x7fc0441066b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:50.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 >> 172.21.15.178:0/958997934 conn(0x7fc0440f0b70 msgr2=0x7fc0440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:50.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 shutdown_connections 2024-01-21T11:16:50.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:50.167+0000 7fc04a254700 1 -- 172.21.15.178:0/958997934 wait complete. 2024-01-21T11:16:50.168 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:51.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:49.671459+0000 mgr.smithi089.ufybqw (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:51.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:50.166057+0000 mon.smithi089 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.178:0/958997934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:51.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:51.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:52.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:51 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:51.671708+0000 mgr.smithi089.ufybqw (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 -- 172.21.15.178:0/1688270887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 msgr2=0x7f56340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 --2- 172.21.15.178:0/1688270887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f56340f66c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f561c009fb0 tx=0x7f561c009b30).stop 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 -- 172.21.15.178:0/1688270887 shutdown_connections 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 --2- 172.21.15.178:0/1688270887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f56340f66c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 -- 172.21.15.178:0/1688270887 >> 172.21.15.178:0/1688270887 conn(0x7f56340f1ac0 msgr2=0x7f56340f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 -- 172.21.15.178:0/1688270887 shutdown_connections 2024-01-21T11:16:52.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.808+0000 7f56397f7700 1 -- 172.21.15.178:0/1688270887 wait complete. 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.809+0000 7f56397f7700 1 Processor -- start 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.809+0000 7f56397f7700 1 -- start start 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.809+0000 7f56397f7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f5634106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.809+0000 7f56397f7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56341069f0 con 0x7f56340f62a0 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5633fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f5634106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5633fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f5634106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57512/0 (socket says 172.21.15.178:57512) 2024-01-21T11:16:52.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5633fff700 1 -- 172.21.15.178:0/3749727487 learned_addr learned my addr 172.21.15.178:0/3749727487 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5633fff700 1 -- 172.21.15.178:0/3749727487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c005040 con 0x7f56340f62a0 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5633fff700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f5634106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f561c009830 tx=0x7f561c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f561c009320 con 0x7f56340f62a0 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f561c0136c0 con 0x7f56340f62a0 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634106bf0 con 0x7f56340f62a0 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f561c01ee90 con 0x7f56340f62a0 2024-01-21T11:16:52.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.810+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5634107080 con 0x7f56340f62a0 2024-01-21T11:16:52.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.811+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f561c01e420 con 0x7f56340f62a0 2024-01-21T11:16:52.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.811+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5634045ad0 con 0x7f56340f62a0 2024-01-21T11:16:52.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.811+0000 7f5630ff9700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f562002ff10 0x7f56200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:52.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.811+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f561c041040 con 0x7f56340f62a0 2024-01-21T11:16:52.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.811+0000 7f56337fe700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f562002ff10 0x7f56200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:52.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.812+0000 7f56337fe700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f562002ff10 0x7f56200323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f56240096a0 tx=0x7f5624006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:52.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:52.816+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f561c06d2b0 con 0x7f56340f62a0 2024-01-21T11:16:53.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.021+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5634107f30 con 0x7f56340f62a0 2024-01-21T11:16:53.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.022+0000 7f5630ff9700 1 -- 172.21.15.178:0/3749727487 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f561c01c790 con 0x7f56340f62a0 2024-01-21T11:16:53.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:53.022 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f562002ff10 msgr2=0x7f56200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f562002ff10 0x7f56200323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f56240096a0 tx=0x7f5624006b40).stop 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 msgr2=0x7f5634106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f5634106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f561c009830 tx=0x7f561c00b040).stop 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 shutdown_connections 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f562002ff10 0x7f56200323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 --2- 172.21.15.178:0/3749727487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56340f62a0 0x7f5634106500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.023+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 >> 172.21.15.178:0/3749727487 conn(0x7f56340f1ac0 msgr2=0x7f56340f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:53.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.024+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 shutdown_connections 2024-01-21T11:16:53.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:53.024+0000 7f56397f7700 1 -- 172.21.15.178:0/3749727487 wait complete. 2024-01-21T11:16:53.025 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:53.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:53.022574+0000 mon.smithi089 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.178:0/3749727487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:54.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:54.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:54.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:53.672026+0000 mgr.smithi089.ufybqw (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 -- 172.21.15.178:0/681474410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 msgr2=0x7efdbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 --2- 172.21.15.178:0/681474410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc0f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7efdac009fb0 tx=0x7efdac009b30).stop 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 -- 172.21.15.178:0/681474410 shutdown_connections 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 --2- 172.21.15.178:0/681474410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc0f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 -- 172.21.15.178:0/681474410 >> 172.21.15.178:0/681474410 conn(0x7efdbc0f0b70 msgr2=0x7efdbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 -- 172.21.15.178:0/681474410 shutdown_connections 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.570+0000 7efdc0fd7700 1 -- 172.21.15.178:0/681474410 wait complete. 2024-01-21T11:16:55.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdc0fd7700 1 Processor -- start 2024-01-21T11:16:55.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdc0fd7700 1 -- start start 2024-01-21T11:16:55.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdc0fd7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:55.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdc0fd7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc105af0 con 0x7efdbc0f5410 2024-01-21T11:16:55.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:55.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57530/0 (socket says 172.21.15.178:57530) 2024-01-21T11:16:55.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.571+0000 7efdba59c700 1 -- 172.21.15.178:0/719991730 learned_addr learned my addr 172.21.15.178:0/719991730 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:55.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdba59c700 1 -- 172.21.15.178:0/719991730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac005040 con 0x7efdbc0f5410 2024-01-21T11:16:55.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdba59c700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc105600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efdac013510 tx=0x7efdac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:55.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdac009320 con 0x7efdbc0f5410 2024-01-21T11:16:55.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc105cf0 con 0x7efdbc0f5410 2024-01-21T11:16:55.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdac013f00 con 0x7efdbc0f5410 2024-01-21T11:16:55.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdac014d90 con 0x7efdbc0f5410 2024-01-21T11:16:55.575 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.572+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc106180 con 0x7efdbc0f5410 2024-01-21T11:16:55.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.573+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdac0263a0 con 0x7efdbc0f5410 2024-01-21T11:16:55.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.573+0000 7efdb2ffd700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efda802ff60 0x7efda8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:55.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.573+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdbc045ad0 con 0x7efdbc0f5410 2024-01-21T11:16:55.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.573+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdac041370 con 0x7efdbc0f5410 2024-01-21T11:16:55.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.573+0000 7efdb9d9b700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efda802ff60 0x7efda8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:55.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.573+0000 7efdb9d9b700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efda802ff60 0x7efda8032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7efda40096a0 tx=0x7efda4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:55.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.578+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdac01cad0 con 0x7efdbc0f5410 2024-01-21T11:16:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.784+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdbc0faf50 con 0x7efdbc0f5410 2024-01-21T11:16:55.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.785+0000 7efdb2ffd700 1 -- 172.21.15.178:0/719991730 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdac00e2a0 con 0x7efdbc0f5410 2024-01-21T11:16:55.785 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:55.785 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:55.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efda802ff60 msgr2=0x7efda8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:55.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efda802ff60 0x7efda8032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7efda40096a0 tx=0x7efda4006b40).stop 2024-01-21T11:16:55.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 msgr2=0x7efdbc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc105600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efdac013510 tx=0x7efdac00b040).stop 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 shutdown_connections 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efda802ff60 0x7efda8032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 --2- 172.21.15.178:0/719991730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdbc0f5410 0x7efdbc105600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 >> 172.21.15.178:0/719991730 conn(0x7efdbc0f0b70 msgr2=0x7efdbc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 shutdown_connections 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:55.786+0000 7efdc0fd7700 1 -- 172.21.15.178:0/719991730 wait complete. 2024-01-21T11:16:55.787 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:56.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:55.672365+0000 mgr.smithi089.ufybqw (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:56.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:55.785405+0000 mon.smithi089 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.178:0/719991730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:57.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:57.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 -- 172.21.15.178:0/3754484880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 msgr2=0x7f13340f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 --2- 172.21.15.178:0/3754484880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f13340f4be0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f131c009fb0 tx=0x7f131c009b30).stop 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 -- 172.21.15.178:0/3754484880 shutdown_connections 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 --2- 172.21.15.178:0/3754484880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f13340f4be0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 -- 172.21.15.178:0/3754484880 >> 172.21.15.178:0/3754484880 conn(0x7f13340f02e0 msgr2=0x7f13340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 -- 172.21.15.178:0/3754484880 shutdown_connections 2024-01-21T11:16:58.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.327+0000 7f1339d26700 1 -- 172.21.15.178:0/3754484880 wait complete. 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f1339d26700 1 Processor -- start 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f1339d26700 1 -- start start 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f1339d26700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f1334106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f1339d26700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334106bf0 con 0x7f13340f47c0 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f13337fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f1334106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f13337fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f1334106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57558/0 (socket says 172.21.15.178:57558) 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.328+0000 7f13337fe700 1 -- 172.21.15.178:0/927393189 learned_addr learned my addr 172.21.15.178:0/927393189 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f13337fe700 1 -- 172.21.15.178:0/927393189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c005040 con 0x7f13340f47c0 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f13337fe700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f1334106700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f131c013510 tx=0x7f131c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f131c009320 con 0x7f13340f47c0 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1334106df0 con 0x7f13340f47c0 2024-01-21T11:16:58.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f131c013f00 con 0x7f13340f47c0 2024-01-21T11:16:58.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f131c014d90 con 0x7f13340f47c0 2024-01-21T11:16:58.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.329+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1334107280 con 0x7f13340f47c0 2024-01-21T11:16:58.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.330+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f131c014420 con 0x7f13340f47c0 2024-01-21T11:16:58.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.330+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1334045ad0 con 0x7f13340f47c0 2024-01-21T11:16:58.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.330+0000 7f132bfff700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f132002ffb0 0x7f1320032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:16:58.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.330+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f131c040fd0 con 0x7f13340f47c0 2024-01-21T11:16:58.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.330+0000 7f1332ffd700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f132002ffb0 0x7f1320032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:16:58.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.331+0000 7f1332ffd700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f132002ffb0 0x7f1320032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f13240096a0 tx=0x7f1324006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:16:58.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.335+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f131c06d2b0 con 0x7f13340f47c0 2024-01-21T11:16:58.539 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.539+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1334108130 con 0x7f13340f47c0 2024-01-21T11:16:58.539 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.539+0000 7f132bfff700 1 -- 172.21.15.178:0/927393189 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f131c0095f0 con 0x7f13340f47c0 2024-01-21T11:16:58.539 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:16:58.539 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f132002ffb0 msgr2=0x7f1320032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f132002ffb0 0x7f1320032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f13240096a0 tx=0x7f1324006b40).stop 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 msgr2=0x7f1334106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f1334106700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f131c013510 tx=0x7f131c00b040).stop 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 shutdown_connections 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f132002ffb0 0x7f1320032470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 --2- 172.21.15.178:0/927393189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13340f47c0 0x7f1334106700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 >> 172.21.15.178:0/927393189 conn(0x7f13340f02e0 msgr2=0x7f13340f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:16:58.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 shutdown_connections 2024-01-21T11:16:58.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:16:58.541+0000 7f1339d26700 1 -- 172.21.15.178:0/927393189 wait complete. 2024-01-21T11:16:58.542 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:16:58.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:57.672725+0000 mgr.smithi089.ufybqw (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:16:58.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:16:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:16:58.540273+0000 mon.smithi089 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.178:0/927393189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:16:59.926 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:16:59.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:00.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:16:59.673059+0000 mgr.smithi089.ufybqw (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.097+0000 7f7505bc8700 1 -- 172.21.15.178:0/204111704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 msgr2=0x7f75000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.097+0000 7f7505bc8700 1 --2- 172.21.15.178:0/204111704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75000f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f74f4009fb0 tx=0x7f74f4009b30).stop 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.097+0000 7f7505bc8700 1 -- 172.21.15.178:0/204111704 shutdown_connections 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.097+0000 7f7505bc8700 1 --2- 172.21.15.178:0/204111704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75000f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.097+0000 7f7505bc8700 1 -- 172.21.15.178:0/204111704 >> 172.21.15.178:0/204111704 conn(0x7f75000f09a0 msgr2=0x7f75000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.098+0000 7f7505bc8700 1 -- 172.21.15.178:0/204111704 shutdown_connections 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.098+0000 7f7505bc8700 1 -- 172.21.15.178:0/204111704 wait complete. 2024-01-21T11:17:01.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.098+0000 7f7505bc8700 1 Processor -- start 2024-01-21T11:17:01.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.098+0000 7f7505bc8700 1 -- start start 2024-01-21T11:17:01.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7505bc8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75001053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:01.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7505bc8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75001058a0 con 0x7f75000f51a0 2024-01-21T11:17:01.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7504bc6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75001053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:01.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7504bc6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75001053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57574/0 (socket says 172.21.15.178:57574) 2024-01-21T11:17:01.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7504bc6700 1 -- 172.21.15.178:0/234273137 learned_addr learned my addr 172.21.15.178:0/234273137 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7504bc6700 1 -- 172.21.15.178:0/234273137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4005040 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7504bc6700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75001053b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f74f4009830 tx=0x7f74f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74f4009320 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7500105aa0 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.099+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74f40136c0 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.100+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74f401ee90 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.100+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7500105f30 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.100+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74f4014440 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.100+0000 7f74fd7fa700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f74f002ff10 0x7f74f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.100+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74f4040fe0 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.101+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7500059100 con 0x7f75000f51a0 2024-01-21T11:17:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.101+0000 7f74fffff700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f74f002ff10 0x7f74f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:01.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.101+0000 7f74fffff700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f74f002ff10 0x7f74f00323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f74ec0096a0 tx=0x7f74ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:01.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.105+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74f40273f0 con 0x7f75000f51a0 2024-01-21T11:17:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.314+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75000fabe0 con 0x7f75000f51a0 2024-01-21T11:17:01.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.314+0000 7f74fd7fa700 1 -- 172.21.15.178:0/234273137 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74f401c320 con 0x7f75000f51a0 2024-01-21T11:17:01.314 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:01.315 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:01.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f74f002ff10 msgr2=0x7f74f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:01.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f74f002ff10 0x7f74f00323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f74ec0096a0 tx=0x7f74ec006b40).stop 2024-01-21T11:17:01.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 msgr2=0x7f75001053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:01.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75001053b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f74f4009830 tx=0x7f74f400b040).stop 2024-01-21T11:17:01.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 shutdown_connections 2024-01-21T11:17:01.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f74f002ff10 0x7f74f00323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:01.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 --2- 172.21.15.178:0/234273137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75000f51a0 0x7f75001053b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:01.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 >> 172.21.15.178:0/234273137 conn(0x7f75000f09a0 msgr2=0x7f75000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:01.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 shutdown_connections 2024-01-21T11:17:01.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:01.316+0000 7f7505bc8700 1 -- 172.21.15.178:0/234273137 wait complete. 2024-01-21T11:17:01.317 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:02.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:01.315284+0000 mon.smithi089 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.178:0/234273137' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:02.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:02.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:01.673389+0000 mgr.smithi089.ufybqw (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/2494156094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 msgr2=0x7fe7a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 --2- 172.21.15.178:0/2494156094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a00f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fe788009fb0 tx=0x7fe788009b30).stop 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/2494156094 shutdown_connections 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 --2- 172.21.15.178:0/2494156094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a00f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/2494156094 >> 172.21.15.178:0/2494156094 conn(0x7fe7a00f0b70 msgr2=0x7fe7a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/2494156094 shutdown_connections 2024-01-21T11:17:03.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/2494156094 wait complete. 2024-01-21T11:17:03.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.855+0000 7fe7a5dc9700 1 Processor -- start 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe7a5dc9700 1 -- start start 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe7a5dc9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe7a5dc9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a0060220 con 0x7fe7a00f5410 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe79f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe79f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36814/0 (socket says 172.21.15.178:36814) 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe79f7fe700 1 -- 172.21.15.178:0/3077561126 learned_addr learned my addr 172.21.15.178:0/3077561126 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe79f7fe700 1 -- 172.21.15.178:0/3077561126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe788005040 con 0x7fe7a00f5410 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.856+0000 7fe79f7fe700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a0109450 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe788013510 tx=0x7fe78800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe788009320 con 0x7fe7a00f5410 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe788013f00 con 0x7fe7a00f5410 2024-01-21T11:17:03.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a0060420 con 0x7fe7a00f5410 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe788014d90 con 0x7fe7a00f5410 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a0109b00 con 0x7fe7a00f5410 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe788014420 con 0x7fe7a00f5410 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a0059100 con 0x7fe7a00f5410 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe797fff700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe78c02ff60 0x7fe78c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.857+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe788041070 con 0x7fe7a00f5410 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.858+0000 7fe79effd700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe78c02ff60 0x7fe78c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:03.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.858+0000 7fe79effd700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe78c02ff60 0x7fe78c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe7900096a0 tx=0x7fe790006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:03.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:03.862+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7880095f0 con 0x7fe7a00f5410 2024-01-21T11:17:04.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.070+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7a0045ad0 con 0x7fe7a00f5410 2024-01-21T11:17:04.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.070+0000 7fe797fff700 1 -- 172.21.15.178:0/3077561126 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe78801db70 con 0x7fe7a00f5410 2024-01-21T11:17:04.072 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:04.072 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe78c02ff60 msgr2=0x7fe78c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe78c02ff60 0x7fe78c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe7900096a0 tx=0x7fe790006b40).stop 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 msgr2=0x7fe7a0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a0109450 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe788013510 tx=0x7fe78800b040).stop 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 shutdown_connections 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe78c02ff60 0x7fe78c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:04.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 --2- 172.21.15.178:0/3077561126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a00f5410 0x7fe7a0109450 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 >> 172.21.15.178:0/3077561126 conn(0x7fe7a00f0b70 msgr2=0x7fe7a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 shutdown_connections 2024-01-21T11:17:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:04.072+0000 7fe7a5dc9700 1 -- 172.21.15.178:0/3077561126 wait complete. 2024-01-21T11:17:04.074 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:05.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:03.673745+0000 mgr.smithi089.ufybqw (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:05.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:04.072145+0000 mon.smithi089 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.178:0/3077561126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:05.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:05.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 -- 172.21.15.178:0/581102984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 msgr2=0x7f32c00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 --2- 172.21.15.178:0/581102984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f9190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f32b0009fb0 tx=0x7f32b0009b30).stop 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 -- 172.21.15.178:0/581102984 shutdown_connections 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 --2- 172.21.15.178:0/581102984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f9190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 -- 172.21.15.178:0/581102984 >> 172.21.15.178:0/581102984 conn(0x7f32c00f0b70 msgr2=0x7f32c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 -- 172.21.15.178:0/581102984 shutdown_connections 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.634+0000 7f32c69a5700 1 -- 172.21.15.178:0/581102984 wait complete. 2024-01-21T11:17:06.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32c69a5700 1 Processor -- start 2024-01-21T11:17:06.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32c69a5700 1 -- start start 2024-01-21T11:17:06.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32c69a5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:06.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32c69a5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c00f5090 con 0x7f32c00601e0 2024-01-21T11:17:06.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32bffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:06.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32bffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36842/0 (socket says 172.21.15.178:36842) 2024-01-21T11:17:06.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.635+0000 7f32bffff700 1 -- 172.21.15.178:0/2024227962 learned_addr learned my addr 172.21.15.178:0/2024227962 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:06.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32bffff700 1 -- 172.21.15.178:0/2024227962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0005040 con 0x7f32c00601e0 2024-01-21T11:17:06.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32bffff700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f6c10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f32b0004dd0 tx=0x7f32b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:06.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32b0018070 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c00f5230 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32b0013410 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32b001c3c0 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.636+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c00f56c0 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.637+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32b001c520 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.637+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c0059100 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.637+0000 7f32bcff9700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32a802ffb0 0x7f32a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.637+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32b00413d0 con 0x7f32c00601e0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.637+0000 7f32bf7fe700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32a802ffb0 0x7f32a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:06.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.638+0000 7f32bf7fe700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32a802ffb0 0x7f32a8032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f32b40096a0 tx=0x7f32b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:06.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.642+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32b001d420 con 0x7f32c00601e0 2024-01-21T11:17:06.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.851+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32c00fbf60 con 0x7f32c00601e0 2024-01-21T11:17:06.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.852+0000 7f32bcff9700 1 -- 172.21.15.178:0/2024227962 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32b001b370 con 0x7f32c00601e0 2024-01-21T11:17:06.853 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:06.853 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:06.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.853+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32a802ffb0 msgr2=0x7f32a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:06.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.853+0000 7f32c69a5700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32a802ffb0 0x7f32a8032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f32b40096a0 tx=0x7f32b4006b40).stop 2024-01-21T11:17:06.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.853+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 msgr2=0x7f32c00f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.853+0000 7f32c69a5700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f6c10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f32b0004dd0 tx=0x7f32b0009b30).stop 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.853+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 shutdown_connections 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.854+0000 7f32c69a5700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32a802ffb0 0x7f32a8032470 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.854+0000 7f32c69a5700 1 --2- 172.21.15.178:0/2024227962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c00601e0 0x7f32c00f6c10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.854+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 >> 172.21.15.178:0/2024227962 conn(0x7f32c00f0b70 msgr2=0x7f32c00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.854+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 shutdown_connections 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:06.854+0000 7f32c69a5700 1 -- 172.21.15.178:0/2024227962 wait complete. 2024-01-21T11:17:06.855 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:07.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:05.674034+0000 mgr.smithi089.ufybqw (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:08.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:07 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:06.853535+0000 mon.smithi089 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.178:0/2024227962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:08.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:08.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:09.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:07.674299+0000 mgr.smithi089.ufybqw (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:09.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.384+0000 7f61dea60700 1 -- 172.21.15.178:0/2513235670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 msgr2=0x7f61d80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:09.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.384+0000 7f61dea60700 1 --2- 172.21.15.178:0/2513235670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d80f68a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f61c8009fb0 tx=0x7f61c8009b30).stop 2024-01-21T11:17:09.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.384+0000 7f61dea60700 1 -- 172.21.15.178:0/2513235670 shutdown_connections 2024-01-21T11:17:09.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.384+0000 7f61dea60700 1 --2- 172.21.15.178:0/2513235670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d80f68a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:09.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.384+0000 7f61dea60700 1 -- 172.21.15.178:0/2513235670 >> 172.21.15.178:0/2513235670 conn(0x7f61d80f1c00 msgr2=0x7f61d80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.385+0000 7f61dea60700 1 -- 172.21.15.178:0/2513235670 shutdown_connections 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.385+0000 7f61dea60700 1 -- 172.21.15.178:0/2513235670 wait complete. 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.385+0000 7f61dea60700 1 Processor -- start 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.385+0000 7f61dea60700 1 -- start start 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.385+0000 7f61dea60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.385+0000 7f61dea60700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d8106bc0 con 0x7f61d80f6480 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36872/0 (socket says 172.21.15.178:36872) 2024-01-21T11:17:09.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d7fff700 1 -- 172.21.15.178:0/4065536488 learned_addr learned my addr 172.21.15.178:0/4065536488 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:09.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d7fff700 1 -- 172.21.15.178:0/4065536488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8005040 con 0x7f61d80f6480 2024-01-21T11:17:09.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d7fff700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d81066d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f61c8014000 tx=0x7f61c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c80095f0 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d8106dc0 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61c8007e70 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c8014ea0 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.386+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d8107250 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.387+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61c8025070 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.387+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d8045ad0 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.387+0000 7f61d4ff9700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.387+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61c80409f0 con 0x7f61d80f6480 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.387+0000 7f61d77fe700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:09.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.388+0000 7f61d77fe700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f61cc0096a0 tx=0x7f61cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:09.393 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.392+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61c800f4e0 con 0x7f61d80f6480 2024-01-21T11:17:09.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.597+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61d8108100 con 0x7f61d80f6480 2024-01-21T11:17:09.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.598+0000 7f61d4ff9700 1 -- 172.21.15.178:0/4065536488 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61c800f300 con 0x7f61d80f6480 2024-01-21T11:17:09.599 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:09.599 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.599+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 msgr2=0x7f61c0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.599+0000 7f61dea60700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f61cc0096a0 tx=0x7f61cc006b40).stop 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.599+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 msgr2=0x7f61d81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d81066d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f61c8014000 tx=0x7f61c800b040).stop 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 shutdown_connections 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 --2- 172.21.15.178:0/4065536488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d80f6480 0x7f61d81066d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 >> 172.21.15.178:0/4065536488 conn(0x7f61d80f1c00 msgr2=0x7f61d80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 shutdown_connections 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:09.600+0000 7f61dea60700 1 -- 172.21.15.178:0/4065536488 wait complete. 2024-01-21T11:17:09.601 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:10.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:09 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:09.599778+0000 mon.smithi089 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.178:0/4065536488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:10.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:10.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:11.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:09.674653+0000 mgr.smithi089.ufybqw (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 -- 172.21.15.178:0/2854994047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 msgr2=0x7f2a6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 --2- 172.21.15.178:0/2854994047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c0f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f2a5c009fb0 tx=0x7f2a5c009b30).stop 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 -- 172.21.15.178:0/2854994047 shutdown_connections 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 --2- 172.21.15.178:0/2854994047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c0f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 -- 172.21.15.178:0/2854994047 >> 172.21.15.178:0/2854994047 conn(0x7f2a6c0f0b70 msgr2=0x7f2a6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 -- 172.21.15.178:0/2854994047 shutdown_connections 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.133+0000 7f2a744d4700 1 -- 172.21.15.178:0/2854994047 wait complete. 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a744d4700 1 Processor -- start 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a744d4700 1 -- start start 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a744d4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a744d4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c105b20 con 0x7f2a6c0f5410 2024-01-21T11:17:12.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a72270700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a72270700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36900/0 (socket says 172.21.15.178:36900) 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.134+0000 7f2a72270700 1 -- 172.21.15.178:0/509224621 learned_addr learned my addr 172.21.15.178:0/509224621 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a72270700 1 -- 172.21.15.178:0/509224621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c005040 con 0x7f2a6c0f5410 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a72270700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c105630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f2a5c013510 tx=0x7f2a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a5c009320 con 0x7f2a6c0f5410 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c105d20 con 0x7f2a6c0f5410 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a5c013f00 con 0x7f2a6c0f5410 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a5c014d90 con 0x7f2a6c0f5410 2024-01-21T11:17:12.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.135+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6c1061b0 con 0x7f2a6c0f5410 2024-01-21T11:17:12.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.136+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a5c0263a0 con 0x7f2a6c0f5410 2024-01-21T11:17:12.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.136+0000 7f2a62ffd700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2a5802ff60 0x7f2a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:12.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.136+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a5c041020 con 0x7f2a6c0f5410 2024-01-21T11:17:12.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.136+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6c045ad0 con 0x7f2a6c0f5410 2024-01-21T11:17:12.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.136+0000 7f2a71a6f700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2a5802ff60 0x7f2a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:12.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.137+0000 7f2a71a6f700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2a5802ff60 0x7f2a58032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2a680096a0 tx=0x7f2a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:12.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.141+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a5c012920 con 0x7f2a6c0f5410 2024-01-21T11:17:12.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.347+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a6c0faf50 con 0x7f2a6c0f5410 2024-01-21T11:17:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.347+0000 7f2a62ffd700 1 -- 172.21.15.178:0/509224621 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a5c02b020 con 0x7f2a6c0f5410 2024-01-21T11:17:12.348 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:12.348 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2a5802ff60 msgr2=0x7f2a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2a5802ff60 0x7f2a58032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2a680096a0 tx=0x7f2a68006b40).stop 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 msgr2=0x7f2a6c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c105630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f2a5c013510 tx=0x7f2a5c00b040).stop 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 shutdown_connections 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2a5802ff60 0x7f2a58032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 --2- 172.21.15.178:0/509224621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6c0f5410 0x7f2a6c105630 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 >> 172.21.15.178:0/509224621 conn(0x7f2a6c0f0b70 msgr2=0x7f2a6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 shutdown_connections 2024-01-21T11:17:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:12.349+0000 7f2a744d4700 1 -- 172.21.15.178:0/509224621 wait complete. 2024-01-21T11:17:12.351 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:13.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:11.674940+0000 mgr.smithi089.ufybqw (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:13.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:12.349036+0000 mon.smithi089 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.178:0/509224621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:13.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:13.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:14.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 -- 172.21.15.178:0/551131633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 msgr2=0x7f2bbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 --2- 172.21.15.178:0/551131633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc0f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f2bb0009fb0 tx=0x7f2bb0009b30).stop 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 -- 172.21.15.178:0/551131633 shutdown_connections 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 --2- 172.21.15.178:0/551131633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc0f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 -- 172.21.15.178:0/551131633 >> 172.21.15.178:0/551131633 conn(0x7f2bbc0f0b70 msgr2=0x7f2bbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 -- 172.21.15.178:0/551131633 shutdown_connections 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.949+0000 7f2bc0c53700 1 -- 172.21.15.178:0/551131633 wait complete. 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bc0c53700 1 Processor -- start 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bc0c53700 1 -- start start 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bc0c53700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bc0c53700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc060220 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:54354/0 (socket says 172.21.15.178:54354) 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bba59c700 1 -- 172.21.15.178:0/1173310546 learned_addr learned my addr 172.21.15.178:0/1173310546 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.950+0000 7f2bba59c700 1 -- 172.21.15.178:0/1173310546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb0005040 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2bba59c700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc109450 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2bb0013510 tx=0x7f2bb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb0009320 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc060420 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb0013f00 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb0014d90 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc109b00 con 0x7f2bbc0f5410 2024-01-21T11:17:14.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.951+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bb0014420 con 0x7f2bbc0f5410 2024-01-21T11:17:14.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.952+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc059100 con 0x7f2bbc0f5410 2024-01-21T11:17:14.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.952+0000 7f2baaffd700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2ba402ff60 0x7f2ba4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:14.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.952+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bb0041080 con 0x7f2bbc0f5410 2024-01-21T11:17:14.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.952+0000 7f2bb9d9b700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2ba402ff60 0x7f2ba4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:14.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.952+0000 7f2bb9d9b700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2ba402ff60 0x7f2ba4032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2bac0096a0 tx=0x7f2bac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:14.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:14.956+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb00095f0 con 0x7f2bbc0f5410 2024-01-21T11:17:15.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.165+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bbc045ad0 con 0x7f2bbc0f5410 2024-01-21T11:17:15.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.166+0000 7f2baaffd700 1 -- 172.21.15.178:0/1173310546 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bb001db70 con 0x7f2bbc0f5410 2024-01-21T11:17:15.167 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:15.167 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:15.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2ba402ff60 msgr2=0x7f2ba4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:15.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2ba402ff60 0x7f2ba4032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2bac0096a0 tx=0x7f2bac006b40).stop 2024-01-21T11:17:15.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 msgr2=0x7f2bbc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc109450 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2bb0013510 tx=0x7f2bb000b040).stop 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 shutdown_connections 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2ba402ff60 0x7f2ba4032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 --2- 172.21.15.178:0/1173310546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc0f5410 0x7f2bbc109450 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 >> 172.21.15.178:0/1173310546 conn(0x7f2bbc0f0b70 msgr2=0x7f2bbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 shutdown_connections 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:15.167+0000 7f2bc0c53700 1 -- 172.21.15.178:0/1173310546 wait complete. 2024-01-21T11:17:15.169 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:15.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:13.675202+0000 mgr.smithi089.ufybqw (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:15.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:13.774110+0000 mon.smithi089 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:17:15.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:13.783426+0000 mon.smithi089 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:17:16.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:15.167520+0000 mon.smithi089 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.178:0/1173310546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:16.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:16.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:17.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:15.675466+0000 mgr.smithi089.ufybqw (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:17.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 -- 172.21.15.178:0/1969399490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 msgr2=0x7f39140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:17.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 --2- 172.21.15.178:0/1969399490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39140f66c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f3910009fb0 tx=0x7f3910009b30).stop 2024-01-21T11:17:17.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 -- 172.21.15.178:0/1969399490 shutdown_connections 2024-01-21T11:17:17.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 --2- 172.21.15.178:0/1969399490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39140f66c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:17.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 -- 172.21.15.178:0/1969399490 >> 172.21.15.178:0/1969399490 conn(0x7f39140f1aa0 msgr2=0x7f39140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:17.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 -- 172.21.15.178:0/1969399490 shutdown_connections 2024-01-21T11:17:17.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.726+0000 7f391ac9e700 1 -- 172.21.15.178:0/1969399490 wait complete. 2024-01-21T11:17:17.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.727+0000 7f391ac9e700 1 Processor -- start 2024-01-21T11:17:17.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.727+0000 7f391ac9e700 1 -- start start 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.727+0000 7f391ac9e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39141064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.727+0000 7f391ac9e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39141069a0 con 0x7f39140f62a0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f3919c9c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39141064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f3919c9c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39141064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:54382/0 (socket says 172.21.15.178:54382) 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f3919c9c700 1 -- 172.21.15.178:0/1419429731 learned_addr learned my addr 172.21.15.178:0/1419429731 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f3919c9c700 1 -- 172.21.15.178:0/1419429731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910005040 con 0x7f39140f62a0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f3919c9c700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39141064b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3910013510 tx=0x7f391000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3910009320 con 0x7f39140f62a0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3914106ba0 con 0x7f39140f62a0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3910013f00 con 0x7f39140f62a0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3910014dc0 con 0x7f39140f62a0 2024-01-21T11:17:17.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.728+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3914107030 con 0x7f39140f62a0 2024-01-21T11:17:17.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.729+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39100095f0 con 0x7f39140f62a0 2024-01-21T11:17:17.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.729+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3914045ad0 con 0x7f39140f62a0 2024-01-21T11:17:17.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.729+0000 7f390a7fc700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f390002ff60 0x7f3900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:17.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.729+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39100410c0 con 0x7f39140f62a0 2024-01-21T11:17:17.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.730+0000 7f391949b700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f390002ff60 0x7f3900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:17.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.730+0000 7f391949b700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f390002ff60 0x7f3900032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f39040096a0 tx=0x7f3904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:17.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.734+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3910014420 con 0x7f39140f62a0 2024-01-21T11:17:17.940 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.940+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3914107ee0 con 0x7f39140f62a0 2024-01-21T11:17:17.941 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.941+0000 7f390a7fc700 1 -- 172.21.15.178:0/1419429731 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3910010230 con 0x7f39140f62a0 2024-01-21T11:17:17.941 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:17.941 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f390002ff60 msgr2=0x7f3900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f390002ff60 0x7f3900032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f39040096a0 tx=0x7f3904006b40).stop 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 msgr2=0x7f39141064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39141064b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3910013510 tx=0x7f391000b040).stop 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 shutdown_connections 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f390002ff60 0x7f3900032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:17.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 --2- 172.21.15.178:0/1419429731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39140f62a0 0x7f39141064b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:17.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 >> 172.21.15.178:0/1419429731 conn(0x7f39140f1aa0 msgr2=0x7f39140f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:17.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 shutdown_connections 2024-01-21T11:17:17.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:17.942+0000 7f391ac9e700 1 -- 172.21.15.178:0/1419429731 wait complete. 2024-01-21T11:17:17.944 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:18.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:17.675818+0000 mgr.smithi089.ufybqw (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:19.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:17.942238+0000 mon.smithi089 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.178:0/1419429731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:19.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:19.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:20.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 -- 172.21.15.178:0/2009534216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 msgr2=0x7f36e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:20.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 --2- 172.21.15.178:0/2009534216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e40f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f36d4009fb0 tx=0x7f36d4009b30).stop 2024-01-21T11:17:20.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 -- 172.21.15.178:0/2009534216 shutdown_connections 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 --2- 172.21.15.178:0/2009534216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e40f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 -- 172.21.15.178:0/2009534216 >> 172.21.15.178:0/2009534216 conn(0x7f36e40f09a0 msgr2=0x7f36e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 -- 172.21.15.178:0/2009534216 shutdown_connections 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 -- 172.21.15.178:0/2009534216 wait complete. 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 Processor -- start 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.508+0000 7f36e91e5700 1 -- start start 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e91e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:20.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e91e5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e4106990 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:54412/0 (socket says 172.21.15.178:54412) 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e3fff700 1 -- 172.21.15.178:0/1021788902 learned_addr learned my addr 172.21.15.178:0/1021788902 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e3fff700 1 -- 172.21.15.178:0/1021788902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4005040 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.509+0000 7f36e3fff700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e41064a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f36d4013510 tx=0x7f36d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d4009320 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e4106b90 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36d4013f00 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d4014dc0 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e4107020 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36d4014420 con 0x7f36e40f51a0 2024-01-21T11:17:20.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.510+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e4045ad0 con 0x7f36e40f51a0 2024-01-21T11:17:20.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.511+0000 7f36e0ff9700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36cc02ff60 0x7f36cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:20.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.511+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36d4041110 con 0x7f36e40f51a0 2024-01-21T11:17:20.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.511+0000 7f36e37fe700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36cc02ff60 0x7f36cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:20.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.511+0000 7f36e37fe700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36cc02ff60 0x7f36cc032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f36d80096a0 tx=0x7f36d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:20.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.515+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36d40095f0 con 0x7f36e40f51a0 2024-01-21T11:17:20.721 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.720+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36e4107ed0 con 0x7f36e40f51a0 2024-01-21T11:17:20.721 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e0ff9700 1 -- 172.21.15.178:0/1021788902 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36d401cb10 con 0x7f36e40f51a0 2024-01-21T11:17:20.722 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:20.722 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:20.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36cc02ff60 msgr2=0x7f36cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:20.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36cc02ff60 0x7f36cc032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f36d80096a0 tx=0x7f36d8006b40).stop 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 msgr2=0x7f36e41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e41064a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f36d4013510 tx=0x7f36d400b040).stop 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 shutdown_connections 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36cc02ff60 0x7f36cc032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 --2- 172.21.15.178:0/1021788902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e40f51a0 0x7f36e41064a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.722+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 >> 172.21.15.178:0/1021788902 conn(0x7f36e40f09a0 msgr2=0x7f36e405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.723+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 shutdown_connections 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:20.723+0000 7f36e91e5700 1 -- 172.21.15.178:0/1021788902 wait complete. 2024-01-21T11:17:20.724 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:20.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:19.676165+0000 mgr.smithi089.ufybqw (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:21.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:20.722660+0000 mon.smithi089 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.178:0/1021788902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:22.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:22.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:22.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:21.676519+0000 mgr.smithi089.ufybqw (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 -- 172.21.15.178:0/3137416007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 msgr2=0x7f2e800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 --2- 172.21.15.178:0/3137416007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f90f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2e74009fb0 tx=0x7f2e74009b30).stop 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 -- 172.21.15.178:0/3137416007 shutdown_connections 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 --2- 172.21.15.178:0/3137416007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f90f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 -- 172.21.15.178:0/3137416007 >> 172.21.15.178:0/3137416007 conn(0x7f2e800f0b70 msgr2=0x7f2e800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 -- 172.21.15.178:0/3137416007 shutdown_connections 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.314+0000 7f2e849b0700 1 -- 172.21.15.178:0/3137416007 wait complete. 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e849b0700 1 Processor -- start 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e849b0700 1 -- start start 2024-01-21T11:17:23.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e849b0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:23.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e849b0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e800f4ff0 con 0x7f2e80060280 2024-01-21T11:17:23.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e7e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:23.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e7e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:46978/0 (socket says 172.21.15.178:46978) 2024-01-21T11:17:23.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.315+0000 7f2e7e59c700 1 -- 172.21.15.178:0/3428617625 learned_addr learned my addr 172.21.15.178:0/3428617625 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:23.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e7e59c700 1 -- 172.21.15.178:0/3428617625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e74005040 con 0x7f2e80060280 2024-01-21T11:17:23.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e7e59c700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f6b70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2e74013510 tx=0x7f2e7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e74009320 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e800f5190 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e800f5620 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e74013f00 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.316+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e74014dc0 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.317+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e740263a0 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.317+0000 7f2e6effd700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e6802ffb0 0x7f2e68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.317+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e74040f90 con 0x7f2e80060280 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.317+0000 7f2e7dd9b700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e6802ffb0 0x7f2e68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.317+0000 7f2e7dd9b700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e6802ffb0 0x7f2e68032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f2e700096a0 tx=0x7f2e70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:23.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.319+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e64004fa0 con 0x7f2e80060280 2024-01-21T11:17:23.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.325+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e74012900 con 0x7f2e80060280 2024-01-21T11:17:23.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.529+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e64005670 con 0x7f2e80060280 2024-01-21T11:17:23.530 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.529+0000 7f2e6effd700 1 -- 172.21.15.178:0/3428617625 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e7402b020 con 0x7f2e80060280 2024-01-21T11:17:23.530 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:23.530 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e6802ffb0 msgr2=0x7f2e68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e6802ffb0 0x7f2e68032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f2e700096a0 tx=0x7f2e70006b40).stop 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 msgr2=0x7f2e800f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f6b70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2e74013510 tx=0x7f2e7400b040).stop 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 shutdown_connections 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e6802ffb0 0x7f2e68032470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 --2- 172.21.15.178:0/3428617625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e80060280 0x7f2e800f6b70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 >> 172.21.15.178:0/3428617625 conn(0x7f2e800f0b70 msgr2=0x7f2e800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 shutdown_connections 2024-01-21T11:17:23.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:23.531+0000 7f2e849b0700 1 -- 172.21.15.178:0/3428617625 wait complete. 2024-01-21T11:17:23.533 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:24.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:23 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:23.531121+0000 mon.smithi089 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.178:0/3428617625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:24.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:24.914 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:25.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:23.676844+0000 mgr.smithi089.ufybqw (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:26.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 -- 172.21.15.178:0/1110228876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 msgr2=0x7f60540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:26.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 --2- 172.21.15.178:0/1110228876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60540f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f6044009fb0 tx=0x7f6044009b30).stop 2024-01-21T11:17:26.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 -- 172.21.15.178:0/1110228876 shutdown_connections 2024-01-21T11:17:26.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 --2- 172.21.15.178:0/1110228876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60540f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:26.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 -- 172.21.15.178:0/1110228876 >> 172.21.15.178:0/1110228876 conn(0x7f60540f0b70 msgr2=0x7f60540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 -- 172.21.15.178:0/1110228876 shutdown_connections 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.027+0000 7f605c61d700 1 -- 172.21.15.178:0/1110228876 wait complete. 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605c61d700 1 Processor -- start 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605c61d700 1 -- start start 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605c61d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60541093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605c61d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6054060220 con 0x7f60540f5410 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605a3b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60541093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605a3b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60541093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47008/0 (socket says 172.21.15.178:47008) 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.028+0000 7f605a3b9700 1 -- 172.21.15.178:0/1679725169 learned_addr learned my addr 172.21.15.178:0/1679725169 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f605a3b9700 1 -- 172.21.15.178:0/1679725169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044005040 con 0x7f60540f5410 2024-01-21T11:17:26.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f605a3b9700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60541093f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6044013510 tx=0x7f604400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:26.030 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6044009320 con 0x7f60540f5410 2024-01-21T11:17:26.030 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6054060420 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6044013f00 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6044014d90 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.029+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6054109aa0 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.030+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6044014420 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.030+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6054059100 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.030+0000 7f604affd700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f604002ff60 0x7f6040032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.030+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6044041080 con 0x7f60540f5410 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.030+0000 7f6059bb8700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f604002ff60 0x7f6040032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:26.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.030+0000 7f6059bb8700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f604002ff60 0x7f6040032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f60500096a0 tx=0x7f6050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:26.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.035+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60440095f0 con 0x7f60540f5410 2024-01-21T11:17:26.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.245+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6054045ad0 con 0x7f60540f5410 2024-01-21T11:17:26.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.246+0000 7f604affd700 1 -- 172.21.15.178:0/1679725169 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f604401db70 con 0x7f60540f5410 2024-01-21T11:17:26.246 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:26.246 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:26.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f604002ff60 msgr2=0x7f6040032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:26.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f604002ff60 0x7f6040032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f60500096a0 tx=0x7f6050006b40).stop 2024-01-21T11:17:26.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 msgr2=0x7f60541093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:26.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60541093f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6044013510 tx=0x7f604400b040).stop 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 shutdown_connections 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f604002ff60 0x7f6040032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 --2- 172.21.15.178:0/1679725169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60540f5410 0x7f60541093f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 >> 172.21.15.178:0/1679725169 conn(0x7f60540f0b70 msgr2=0x7f60540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 shutdown_connections 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:26.247+0000 7f605c61d700 1 -- 172.21.15.178:0/1679725169 wait complete. 2024-01-21T11:17:26.248 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:27.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:25.677137+0000 mgr.smithi089.ufybqw (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:27.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:26.246626+0000 mon.smithi089 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.178:0/1679725169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:27.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:27.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 -- 172.21.15.178:0/2370458926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 msgr2=0x7f7d900f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 --2- 172.21.15.178:0/2370458926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d900f5810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f7d80009fb0 tx=0x7f7d80009b30).stop 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 -- 172.21.15.178:0/2370458926 shutdown_connections 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 --2- 172.21.15.178:0/2370458926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d900f5810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 -- 172.21.15.178:0/2370458926 >> 172.21.15.178:0/2370458926 conn(0x7f7d900f0b70 msgr2=0x7f7d900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 -- 172.21.15.178:0/2370458926 shutdown_connections 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.842+0000 7f7d97f4d700 1 -- 172.21.15.178:0/2370458926 wait complete. 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d97f4d700 1 Processor -- start 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d97f4d700 1 -- start start 2024-01-21T11:17:28.843 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d97f4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d901093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:28.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d97f4d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90060240 con 0x7f7d900f53f0 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d95ce9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d901093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d95ce9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d901093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47034/0 (socket says 172.21.15.178:47034) 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.843+0000 7f7d95ce9700 1 -- 172.21.15.178:0/4191561491 learned_addr learned my addr 172.21.15.178:0/4191561491 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d95ce9700 1 -- 172.21.15.178:0/4191561491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80005040 con 0x7f7d900f53f0 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d95ce9700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d901093e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f7d80014000 tx=0x7f7d8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d800095f0 con 0x7f7d900f53f0 2024-01-21T11:17:28.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d90060440 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d80007e70 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d80014ea0 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.844+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d90109a90 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.845+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d80025070 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.845+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d90045ad0 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.845+0000 7f7d867fc700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7d7c02ff60 0x7f7d7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.845+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d80040a30 con 0x7f7d900f53f0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.845+0000 7f7d954e8700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7d7c02ff60 0x7f7d7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:28.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.846+0000 7f7d954e8700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7d7c02ff60 0x7f7d7c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7d8c0096a0 tx=0x7f7d8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:28.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:28.850+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d800079c0 con 0x7f7d900f53f0 2024-01-21T11:17:29.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.056+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d9010a2b0 con 0x7f7d900f53f0 2024-01-21T11:17:29.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.056+0000 7f7d867fc700 1 -- 172.21.15.178:0/4191561491 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d8000f300 con 0x7f7d900f53f0 2024-01-21T11:17:29.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:29.057 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7d7c02ff60 msgr2=0x7f7d7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7d7c02ff60 0x7f7d7c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7d8c0096a0 tx=0x7f7d8c006b40).stop 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 msgr2=0x7f7d901093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d901093e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f7d80014000 tx=0x7f7d8000b040).stop 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 shutdown_connections 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7d7c02ff60 0x7f7d7c032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 --2- 172.21.15.178:0/4191561491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d900f53f0 0x7f7d901093e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:29.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 >> 172.21.15.178:0/4191561491 conn(0x7f7d900f0b70 msgr2=0x7f7d900f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:29.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 shutdown_connections 2024-01-21T11:17:29.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:29.058+0000 7f7d97f4d700 1 -- 172.21.15.178:0/4191561491 wait complete. 2024-01-21T11:17:29.059 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:29.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:27.677364+0000 mgr.smithi089.ufybqw (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:30.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:29.057272+0000 mon.smithi089 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.178:0/4191561491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:30.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:30.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:31.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:29.677702+0000 mgr.smithi089.ufybqw (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 -- 172.21.15.178:0/1944193128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 msgr2=0x7fa43c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 --2- 172.21.15.178:0/1944193128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c0f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fa42c009fb0 tx=0x7fa42c009b30).stop 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 -- 172.21.15.178:0/1944193128 shutdown_connections 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 --2- 172.21.15.178:0/1944193128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c0f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 -- 172.21.15.178:0/1944193128 >> 172.21.15.178:0/1944193128 conn(0x7fa43c0f1c70 msgr2=0x7fa43c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 -- 172.21.15.178:0/1944193128 shutdown_connections 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.618+0000 7fa442789700 1 -- 172.21.15.178:0/1944193128 wait complete. 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.619+0000 7fa442789700 1 Processor -- start 2024-01-21T11:17:31.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.619+0000 7fa442789700 1 -- start start 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.619+0000 7fa442789700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa442789700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c106c20 con 0x7fa43c0f6510 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47054/0 (socket says 172.21.15.178:47054) 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa43bfff700 1 -- 172.21.15.178:0/3489646996 learned_addr learned my addr 172.21.15.178:0/3489646996 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa43bfff700 1 -- 172.21.15.178:0/3489646996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c005040 con 0x7fa43c0f6510 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa43bfff700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fa42c013510 tx=0x7fa42c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa42c009320 con 0x7fa43c0f6510 2024-01-21T11:17:31.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43c106e20 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa42c013f00 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa42c014d90 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.620+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c1072b0 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.621+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa42c0095f0 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.621+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43c045ad0 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.621+0000 7fa438ff9700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.621+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa42c040fc0 con 0x7fa43c0f6510 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.622+0000 7fa43b7fe700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:31.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.622+0000 7fa43b7fe700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa42402ff60 0x7fa424032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa4300096a0 tx=0x7fa430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:31.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.626+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa42c014420 con 0x7fa43c0f6510 2024-01-21T11:17:31.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.830+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa43c0fbf60 con 0x7fa43c0f6510 2024-01-21T11:17:31.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.830+0000 7fa438ff9700 1 -- 172.21.15.178:0/3489646996 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa42c010230 con 0x7fa43c0f6510 2024-01-21T11:17:31.830 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:31.830 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:31.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa42402ff60 msgr2=0x7fa424032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:31.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa42402ff60 0x7fa424032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa4300096a0 tx=0x7fa430006b40).stop 2024-01-21T11:17:31.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 msgr2=0x7fa43c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fa42c013510 tx=0x7fa42c00b040).stop 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 shutdown_connections 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 --2- 172.21.15.178:0/3489646996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c0f6510 0x7fa43c106730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 >> 172.21.15.178:0/3489646996 conn(0x7fa43c0f1c70 msgr2=0x7fa43c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 shutdown_connections 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:31.832+0000 7fa442789700 1 -- 172.21.15.178:0/3489646996 wait complete. 2024-01-21T11:17:31.833 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:33.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:31.678055+0000 mgr.smithi089.ufybqw (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:33.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:31.830956+0000 mon.smithi089 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.178:0/3489646996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:33.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:33.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 -- 172.21.15.178:0/553781866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 msgr2=0x7f030c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 --2- 172.21.15.178:0/553781866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c0f90f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f02fc009fb0 tx=0x7f02fc009b30).stop 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 -- 172.21.15.178:0/553781866 shutdown_connections 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 --2- 172.21.15.178:0/553781866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c0f90f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 -- 172.21.15.178:0/553781866 >> 172.21.15.178:0/553781866 conn(0x7f030c0f0b70 msgr2=0x7f030c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 -- 172.21.15.178:0/553781866 shutdown_connections 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.471+0000 7f03145ee700 1 -- 172.21.15.178:0/553781866 wait complete. 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f03145ee700 1 Processor -- start 2024-01-21T11:17:34.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f03145ee700 1 -- start start 2024-01-21T11:17:34.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f03145ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:34.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f03145ee700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c106bd0 con 0x7f030c060280 2024-01-21T11:17:34.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f031238a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f031238a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58684/0 (socket says 172.21.15.178:58684) 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.472+0000 7f031238a700 1 -- 172.21.15.178:0/3760382994 learned_addr learned my addr 172.21.15.178:0/3760382994 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f031238a700 1 -- 172.21.15.178:0/3760382994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc005040 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f031238a700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c1066e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f02fc013510 tx=0x7f02fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02fc009320 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c106dd0 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02fc013f00 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02fc014dc0 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.473+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c107260 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.474+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02fc01d3d0 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.474+0000 7f0302ffd700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802ff60 0x7f02f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.474+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02fc0410d0 con 0x7f030c060280 2024-01-21T11:17:34.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.474+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c045ad0 con 0x7f030c060280 2024-01-21T11:17:34.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.474+0000 7f0311b89700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802ff60 0x7f02f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:34.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.475+0000 7f0311b89700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802ff60 0x7f02f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f03080096a0 tx=0x7f0308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:34.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.479+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02fc026490 con 0x7f030c060280 2024-01-21T11:17:34.683 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.683+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f030c107fa0 con 0x7f030c060280 2024-01-21T11:17:34.683 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.684+0000 7f0302ffd700 1 -- 172.21.15.178:0/3760382994 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02fc0126f0 con 0x7f030c060280 2024-01-21T11:17:34.683 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:34.684 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802ff60 msgr2=0x7f02f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802ff60 0x7f02f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f03080096a0 tx=0x7f0308006b40).stop 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 msgr2=0x7f030c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c1066e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f02fc013510 tx=0x7f02fc00b040).stop 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 shutdown_connections 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f02f802ff60 0x7f02f8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:34.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 --2- 172.21.15.178:0/3760382994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030c060280 0x7f030c1066e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:34.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 >> 172.21.15.178:0/3760382994 conn(0x7f030c0f0b70 msgr2=0x7f030c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:34.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 shutdown_connections 2024-01-21T11:17:34.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:34.685+0000 7f03145ee700 1 -- 172.21.15.178:0/3760382994 wait complete. 2024-01-21T11:17:34.686 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:35.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:33.678382+0000 mgr.smithi089.ufybqw (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:35.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:34.684383+0000 mon.smithi089 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.178:0/3760382994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:36.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:36.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:37.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:35.678731+0000 mgr.smithi089.ufybqw (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:37.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 -- 172.21.15.178:0/245328424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 msgr2=0x7f57440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:37.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 --2- 172.21.15.178:0/245328424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5738009fb0 tx=0x7f5738009b30).stop 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 -- 172.21.15.178:0/245328424 shutdown_connections 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 --2- 172.21.15.178:0/245328424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 -- 172.21.15.178:0/245328424 >> 172.21.15.178:0/245328424 conn(0x7f57440f0b70 msgr2=0x7f57440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 -- 172.21.15.178:0/245328424 shutdown_connections 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 -- 172.21.15.178:0/245328424 wait complete. 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.246+0000 7f574b94a700 1 Processor -- start 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f574b94a700 1 -- start start 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f574b94a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:37.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f574b94a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440fb820 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f57496e6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f57496e6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58702/0 (socket says 172.21.15.178:58702) 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f57496e6700 1 -- 172.21.15.178:0/2896760201 learned_addr learned my addr 172.21.15.178:0/2896760201 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f57496e6700 1 -- 172.21.15.178:0/2896760201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738005040 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.247+0000 7f57496e6700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440fd3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f5738013510 tx=0x7f573800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.248+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5738009320 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.248+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57440fb9c0 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.248+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5738013f00 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.248+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5738014d90 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.248+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57440fbe50 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.249+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5744059100 con 0x7f57440f5410 2024-01-21T11:17:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.249+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57380095f0 con 0x7f57440f5410 2024-01-21T11:17:37.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.249+0000 7f5735ffb700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f573002ffb0 0x7f5730032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:37.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.249+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5738040f90 con 0x7f57440f5410 2024-01-21T11:17:37.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.249+0000 7f5748ee5700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f573002ffb0 0x7f5730032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:37.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.249+0000 7f5748ee5700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f573002ffb0 0x7f5730032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f57400096a0 tx=0x7f5740006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:37.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.253+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5738014420 con 0x7f57440f5410 2024-01-21T11:17:37.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.459+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57440fae60 con 0x7f57440f5410 2024-01-21T11:17:37.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.459+0000 7f5735ffb700 1 -- 172.21.15.178:0/2896760201 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f573801d7f0 con 0x7f57440f5410 2024-01-21T11:17:37.460 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:37.460 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f573002ffb0 msgr2=0x7f5730032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f573002ffb0 0x7f5730032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f57400096a0 tx=0x7f5740006b40).stop 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 msgr2=0x7f57440fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440fd3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f5738013510 tx=0x7f573800b040).stop 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 shutdown_connections 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f573002ffb0 0x7f5730032470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 --2- 172.21.15.178:0/2896760201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57440f5410 0x7f57440fd3a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 >> 172.21.15.178:0/2896760201 conn(0x7f57440f0b70 msgr2=0x7f574405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:37.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 shutdown_connections 2024-01-21T11:17:37.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:37.461+0000 7f574b94a700 1 -- 172.21.15.178:0/2896760201 wait complete. 2024-01-21T11:17:37.462 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:38.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:37 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:37.460156+0000 mon.smithi089 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.178:0/2896760201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:38.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:38.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:39.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:37.679116+0000 mgr.smithi089.ufybqw (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:39.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 -- 172.21.15.178:0/144539697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 msgr2=0x7f42340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 --2- 172.21.15.178:0/144539697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42340f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f4224009fb0 tx=0x7f4224009b30).stop 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 -- 172.21.15.178:0/144539697 shutdown_connections 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 --2- 172.21.15.178:0/144539697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42340f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 -- 172.21.15.178:0/144539697 >> 172.21.15.178:0/144539697 conn(0x7f42340f0b70 msgr2=0x7f42340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 -- 172.21.15.178:0/144539697 shutdown_connections 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.982+0000 7f423c6fa700 1 -- 172.21.15.178:0/144539697 wait complete. 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423c6fa700 1 Processor -- start 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423c6fa700 1 -- start start 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423c6fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42341093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423c6fa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4234060280 con 0x7f42340f5410 2024-01-21T11:17:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423a496700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42341093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423a496700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42341093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:58714/0 (socket says 172.21.15.178:58714) 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423a496700 1 -- 172.21.15.178:0/2450510601 learned_addr learned my addr 172.21.15.178:0/2450510601 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.983+0000 7f423a496700 1 -- 172.21.15.178:0/2450510601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4224005040 con 0x7f42340f5410 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.984+0000 7f423a496700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42341093f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4224013510 tx=0x7f422400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.984+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4224009320 con 0x7f42340f5410 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.984+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4224013f00 con 0x7f42340f5410 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.984+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4234060480 con 0x7f42340f5410 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.984+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4224014d90 con 0x7f42340f5410 2024-01-21T11:17:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.984+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4234109b20 con 0x7f42340f5410 2024-01-21T11:17:39.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.985+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42240263a0 con 0x7f42340f5410 2024-01-21T11:17:39.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.985+0000 7f422affd700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f422002ff60 0x7f4220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:39.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.985+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4234045ad0 con 0x7f42340f5410 2024-01-21T11:17:39.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.985+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4224041390 con 0x7f42340f5410 2024-01-21T11:17:39.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.985+0000 7f4239c95700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f422002ff60 0x7f4220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:39.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.985+0000 7f4239c95700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f422002ff60 0x7f4220032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f42300096a0 tx=0x7f4230006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:39.989 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:39.990+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4224012510 con 0x7f42340f5410 2024-01-21T11:17:40.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.193+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f423410a340 con 0x7f42340f5410 2024-01-21T11:17:40.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.194+0000 7f422affd700 1 -- 172.21.15.178:0/2450510601 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4224012330 con 0x7f42340f5410 2024-01-21T11:17:40.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:40.194 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:40.195 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f422002ff60 msgr2=0x7f4220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:40.195 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f422002ff60 0x7f4220032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f42300096a0 tx=0x7f4230006b40).stop 2024-01-21T11:17:40.195 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 msgr2=0x7f42341093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:40.195 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42341093f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4224013510 tx=0x7f422400b040).stop 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 shutdown_connections 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f422002ff60 0x7f4220032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 --2- 172.21.15.178:0/2450510601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42340f5410 0x7f42341093f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 >> 172.21.15.178:0/2450510601 conn(0x7f42340f0b70 msgr2=0x7f42340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 shutdown_connections 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:40.195+0000 7f423c6fa700 1 -- 172.21.15.178:0/2450510601 wait complete. 2024-01-21T11:17:40.196 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:41.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:39.679441+0000 mgr.smithi089.ufybqw (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:41.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:40.194573+0000 mon.smithi089 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.178:0/2450510601' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:41.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:41.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:42.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 -- 172.21.15.178:0/1590527502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 msgr2=0x7fb5040f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:42.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 --2- 172.21.15.178:0/1590527502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5040f66c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb4f4009fb0 tx=0x7fb4f4009b30).stop 2024-01-21T11:17:42.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 -- 172.21.15.178:0/1590527502 shutdown_connections 2024-01-21T11:17:42.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 --2- 172.21.15.178:0/1590527502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5040f66c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:42.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 -- 172.21.15.178:0/1590527502 >> 172.21.15.178:0/1590527502 conn(0x7fb5040f1aa0 msgr2=0x7fb5040f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:42.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 -- 172.21.15.178:0/1590527502 shutdown_connections 2024-01-21T11:17:42.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.728+0000 7fb508f2c700 1 -- 172.21.15.178:0/1590527502 wait complete. 2024-01-21T11:17:42.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.729+0000 7fb508f2c700 1 Processor -- start 2024-01-21T11:17:42.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.729+0000 7fb508f2c700 1 -- start start 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb508f2c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5041064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb508f2c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5041069a0 con 0x7fb5040f62a0 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb5037fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5041064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb5037fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5041064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36482/0 (socket says 172.21.15.178:36482) 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb5037fe700 1 -- 172.21.15.178:0/161360683 learned_addr learned my addr 172.21.15.178:0/161360683 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb5037fe700 1 -- 172.21.15.178:0/161360683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4005040 con 0x7fb5040f62a0 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb5037fe700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5041064b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb4f4013510 tx=0x7fb4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4009320 con 0x7fb5040f62a0 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb504106ba0 con 0x7fb5040f62a0 2024-01-21T11:17:42.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f4013f00 con 0x7fb5040f62a0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4014dc0 con 0x7fb5040f62a0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.730+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb504107030 con 0x7fb5040f62a0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.731+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4f40095f0 con 0x7fb5040f62a0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.731+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb504045ad0 con 0x7fb5040f62a0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.731+0000 7fb4ebfff700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4ec02ffb0 0x7fb4ec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.731+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4f40410c0 con 0x7fb5040f62a0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.732+0000 7fb502ffd700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4ec02ffb0 0x7fb4ec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:42.733 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.732+0000 7fb502ffd700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4ec02ffb0 0x7fb4ec032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb4f80096a0 tx=0x7fb4f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:42.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.736+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4f4014420 con 0x7fb5040f62a0 2024-01-21T11:17:42.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.939+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb504107ee0 con 0x7fb5040f62a0 2024-01-21T11:17:42.940 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.940+0000 7fb4ebfff700 1 -- 172.21.15.178:0/161360683 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4f4010230 con 0x7fb5040f62a0 2024-01-21T11:17:42.940 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:42.940 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4ec02ffb0 msgr2=0x7fb4ec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4ec02ffb0 0x7fb4ec032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb4f80096a0 tx=0x7fb4f8006b40).stop 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 msgr2=0x7fb5041064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5041064b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb4f4013510 tx=0x7fb4f400b040).stop 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 shutdown_connections 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4ec02ffb0 0x7fb4ec032470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 --2- 172.21.15.178:0/161360683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5040f62a0 0x7fb5041064b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 >> 172.21.15.178:0/161360683 conn(0x7fb5040f1aa0 msgr2=0x7fb5040f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 shutdown_connections 2024-01-21T11:17:42.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:42.942+0000 7fb508f2c700 1 -- 172.21.15.178:0/161360683 wait complete. 2024-01-21T11:17:42.943 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:43.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:41.679755+0000 mgr.smithi089.ufybqw (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:44.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:42.940905+0000 mon.smithi089 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.178:0/161360683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:44.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:44.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:45.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:43.680073+0000 mgr.smithi089.ufybqw (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:45.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 -- 172.21.15.178:0/3573547333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 msgr2=0x7f99b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:45.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 --2- 172.21.15.178:0/3573547333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b00f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f99a4009fb0 tx=0x7f99a4009b30).stop 2024-01-21T11:17:45.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 -- 172.21.15.178:0/3573547333 shutdown_connections 2024-01-21T11:17:45.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 --2- 172.21.15.178:0/3573547333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b00f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:45.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 -- 172.21.15.178:0/3573547333 >> 172.21.15.178:0/3573547333 conn(0x7f99b00f1c70 msgr2=0x7f99b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:45.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 -- 172.21.15.178:0/3573547333 shutdown_connections 2024-01-21T11:17:45.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.428+0000 7f99b7748700 1 -- 172.21.15.178:0/3573547333 wait complete. 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b7748700 1 Processor -- start 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b7748700 1 -- start start 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b7748700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b7748700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0106c20 con 0x7f99b00f6510 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b54e4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b54e4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36492/0 (socket says 172.21.15.178:36492) 2024-01-21T11:17:45.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.429+0000 7f99b54e4700 1 -- 172.21.15.178:0/1805994118 learned_addr learned my addr 172.21.15.178:0/1805994118 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99b54e4700 1 -- 172.21.15.178:0/1805994118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4005040 con 0x7f99b00f6510 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99b54e4700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b0106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f99a4014000 tx=0x7f99a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a40095f0 con 0x7f99b00f6510 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b0106e20 con 0x7f99b00f6510 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99a4007e70 con 0x7f99b00f6510 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a4014ea0 con 0x7f99b00f6510 2024-01-21T11:17:45.431 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.430+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b01072b0 con 0x7f99b00f6510 2024-01-21T11:17:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.431+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99a4025070 con 0x7f99b00f6510 2024-01-21T11:17:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.431+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0045ad0 con 0x7f99b00f6510 2024-01-21T11:17:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.431+0000 7f99a1ffb700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f999c02ffb0 0x7f999c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.431+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99a4040a30 con 0x7f99b00f6510 2024-01-21T11:17:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.432+0000 7f99b4ce3700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f999c02ffb0 0x7f999c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.432+0000 7f99b4ce3700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f999c02ffb0 0x7f999c032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f99ac0096a0 tx=0x7f99ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:45.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.436+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99a40079c0 con 0x7f99b00f6510 2024-01-21T11:17:45.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.645+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99b00fbf60 con 0x7f99b00f6510 2024-01-21T11:17:45.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.646+0000 7f99a1ffb700 1 -- 172.21.15.178:0/1805994118 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f99a400f300 con 0x7f99b00f6510 2024-01-21T11:17:45.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:45.646 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:45.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f999c02ffb0 msgr2=0x7f999c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f999c02ffb0 0x7f999c032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f99ac0096a0 tx=0x7f99ac006b40).stop 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 msgr2=0x7f99b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b0106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f99a4014000 tx=0x7f99a400b040).stop 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 shutdown_connections 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f999c02ffb0 0x7f999c032470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 --2- 172.21.15.178:0/1805994118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00f6510 0x7f99b0106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 >> 172.21.15.178:0/1805994118 conn(0x7f99b00f1c70 msgr2=0x7f99b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 shutdown_connections 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:45.648+0000 7f99b7748700 1 -- 172.21.15.178:0/1805994118 wait complete. 2024-01-21T11:17:45.648 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:46.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:45.646828+0000 mon.smithi089 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.178:0/1805994118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:46.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:45.680327+0000 mgr.smithi089.ufybqw (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:47.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:47.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:48.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.176+0000 7fb638bb0700 1 -- 172.21.15.178:0/1146212832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 msgr2=0x7fb6340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:48.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.176+0000 7fb638bb0700 1 --2- 172.21.15.178:0/1146212832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6340f55c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fb624009fb0 tx=0x7fb624009b30).stop 2024-01-21T11:17:48.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.176+0000 7fb638bb0700 1 -- 172.21.15.178:0/1146212832 shutdown_connections 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.176+0000 7fb638bb0700 1 --2- 172.21.15.178:0/1146212832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6340f55c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.176+0000 7fb638bb0700 1 -- 172.21.15.178:0/1146212832 >> 172.21.15.178:0/1146212832 conn(0x7fb6340f09a0 msgr2=0x7fb6340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.177+0000 7fb638bb0700 1 -- 172.21.15.178:0/1146212832 shutdown_connections 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.177+0000 7fb638bb0700 1 -- 172.21.15.178:0/1146212832 wait complete. 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.177+0000 7fb638bb0700 1 Processor -- start 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.177+0000 7fb638bb0700 1 -- start start 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.177+0000 7fb638bb0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.177+0000 7fb638bb0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341058a0 con 0x7fb6340f51a0 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb6337fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb6337fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36522/0 (socket says 172.21.15.178:36522) 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb6337fe700 1 -- 172.21.15.178:0/1013260468 learned_addr learned my addr 172.21.15.178:0/1013260468 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb6337fe700 1 -- 172.21.15.178:0/1013260468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb624005040 con 0x7fb6340f51a0 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb6337fe700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb624013510 tx=0x7fb62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb624009320 con 0x7fb6340f51a0 2024-01-21T11:17:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb634105aa0 con 0x7fb6340f51a0 2024-01-21T11:17:48.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb624013f00 con 0x7fb6340f51a0 2024-01-21T11:17:48.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb624014dc0 con 0x7fb6340f51a0 2024-01-21T11:17:48.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.178+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb634105eb0 con 0x7fb6340f51a0 2024-01-21T11:17:48.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.179+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb634045ad0 con 0x7fb6340f51a0 2024-01-21T11:17:48.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.179+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6240095f0 con 0x7fb6340f51a0 2024-01-21T11:17:48.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.179+0000 7fb61bfff700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb61c032170 0x7fb61c034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:48.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.179+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb624041000 con 0x7fb6340f51a0 2024-01-21T11:17:48.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.179+0000 7fb632ffd700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb61c032170 0x7fb61c034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:48.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.180+0000 7fb632ffd700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb61c032170 0x7fb61c034630 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb6280096a0 tx=0x7fb628006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:48.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:47.680653+0000 mgr.smithi089.ufybqw (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:48.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.184+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb62401c790 con 0x7fb6340f51a0 2024-01-21T11:17:48.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.388+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb634106d70 con 0x7fb6340f51a0 2024-01-21T11:17:48.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.389+0000 7fb61bfff700 1 -- 172.21.15.178:0/1013260468 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb624010230 con 0x7fb6340f51a0 2024-01-21T11:17:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:48.389 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:48.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.390+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb61c032170 msgr2=0x7fb61c034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:48.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.390+0000 7fb638bb0700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb61c032170 0x7fb61c034630 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb6280096a0 tx=0x7fb628006b40).stop 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.390+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 msgr2=0x7fb6341053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.390+0000 7fb638bb0700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb624013510 tx=0x7fb62400b040).stop 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.390+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 shutdown_connections 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.390+0000 7fb638bb0700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb61c032170 0x7fb61c034630 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.391+0000 7fb638bb0700 1 --2- 172.21.15.178:0/1013260468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.391+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 >> 172.21.15.178:0/1013260468 conn(0x7fb6340f09a0 msgr2=0x7fb6340f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.391+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 shutdown_connections 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:48.391+0000 7fb638bb0700 1 -- 172.21.15.178:0/1013260468 wait complete. 2024-01-21T11:17:48.391 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:49.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:48.389794+0000 mon.smithi089 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.178:0/1013260468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:49.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:49.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:50.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.885+0000 7f69eb5c4700 1 -- 172.21.15.178:0/1349305167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 msgr2=0x7f69e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:50.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.885+0000 7f69eb5c4700 1 --2- 172.21.15.178:0/1349305167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e40f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f69d4009fb0 tx=0x7f69d4009b30).stop 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.885+0000 7f69eb5c4700 1 -- 172.21.15.178:0/1349305167 shutdown_connections 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.885+0000 7f69eb5c4700 1 --2- 172.21.15.178:0/1349305167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e40f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.885+0000 7f69eb5c4700 1 -- 172.21.15.178:0/1349305167 >> 172.21.15.178:0/1349305167 conn(0x7f69e40f0b70 msgr2=0x7f69e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.885+0000 7f69eb5c4700 1 -- 172.21.15.178:0/1349305167 shutdown_connections 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.886+0000 7f69eb5c4700 1 -- 172.21.15.178:0/1349305167 wait complete. 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.886+0000 7f69eb5c4700 1 Processor -- start 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.886+0000 7f69eb5c4700 1 -- start start 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.886+0000 7f69eb5c4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.886+0000 7f69eb5c4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4060240 con 0x7f69e40f5410 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.886+0000 7f69e9360700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69e9360700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36548/0 (socket says 172.21.15.178:36548) 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69e9360700 1 -- 172.21.15.178:0/4020054185 learned_addr learned my addr 172.21.15.178:0/4020054185 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69e9360700 1 -- 172.21.15.178:0/4020054185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4005040 con 0x7f69e40f5410 2024-01-21T11:17:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69e9360700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f69d4013510 tx=0x7f69d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d4009320 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e4060440 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d4013f00 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d4014d90 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.887+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4109ae0 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.888+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69d4014420 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.888+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e4045ad0 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.888+0000 7f69d9ffb700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f69d0030000 0x7f69d00324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.888+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69d4041090 con 0x7f69e40f5410 2024-01-21T11:17:50.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.888+0000 7f69e8b5f700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f69d0030000 0x7f69d00324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:50.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.889+0000 7f69e8b5f700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f69d0030000 0x7f69d00324c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f69e00096a0 tx=0x7f69e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:50.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:50.893+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69d406d2b0 con 0x7f69e40f5410 2024-01-21T11:17:50.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:49.680993+0000 mgr.smithi089.ufybqw (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:51.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.099+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69e40fae60 con 0x7f69e40f5410 2024-01-21T11:17:51.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.100+0000 7f69d9ffb700 1 -- 172.21.15.178:0/4020054185 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69d40095f0 con 0x7f69e40f5410 2024-01-21T11:17:51.100 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:51.100 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f69d0030000 msgr2=0x7f69d00324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f69d0030000 0x7f69d00324c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f69e00096a0 tx=0x7f69e0006b40).stop 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 msgr2=0x7f69e4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f69d4013510 tx=0x7f69d400b040).stop 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 shutdown_connections 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f69d0030000 0x7f69d00324c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 --2- 172.21.15.178:0/4020054185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 >> 172.21.15.178:0/4020054185 conn(0x7f69e40f0b70 msgr2=0x7f69e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 shutdown_connections 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:51.102+0000 7f69eb5c4700 1 -- 172.21.15.178:0/4020054185 wait complete. 2024-01-21T11:17:51.102 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:52.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:51 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:51.100587+0000 mon.smithi089 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.178:0/4020054185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:52.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:52.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:53.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:51.681321+0000 mgr.smithi089.ufybqw (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 -- 172.21.15.178:0/3978229070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 msgr2=0x7fc2640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 --2- 172.21.15.178:0/3978229070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc2640f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fc254009fb0 tx=0x7fc254009b30).stop 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 -- 172.21.15.178:0/3978229070 shutdown_connections 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 --2- 172.21.15.178:0/3978229070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc2640f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 -- 172.21.15.178:0/3978229070 >> 172.21.15.178:0/3978229070 conn(0x7fc2640f0b70 msgr2=0x7fc2640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 -- 172.21.15.178:0/3978229070 shutdown_connections 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.629+0000 7fc26b317700 1 -- 172.21.15.178:0/3978229070 wait complete. 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc26b317700 1 Processor -- start 2024-01-21T11:17:53.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc26b317700 1 -- start start 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc26b317700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc264109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc26b317700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc264060240 con 0x7fc2640f5410 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc2690b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc264109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc2690b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc264109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42330/0 (socket says 172.21.15.178:42330) 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc2690b3700 1 -- 172.21.15.178:0/4061619259 learned_addr learned my addr 172.21.15.178:0/4061619259 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.630+0000 7fc2690b3700 1 -- 172.21.15.178:0/4061619259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc254005040 con 0x7fc2640f5410 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.631+0000 7fc2690b3700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc264109430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fc254013510 tx=0x7fc25400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.631+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc254009320 con 0x7fc2640f5410 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.631+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc264060440 con 0x7fc2640f5410 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.631+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc254013f00 con 0x7fc2640f5410 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.631+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc254014d90 con 0x7fc2640f5410 2024-01-21T11:17:53.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.631+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc264109ae0 con 0x7fc2640f5410 2024-01-21T11:17:53.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.632+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2540095f0 con 0x7fc2640f5410 2024-01-21T11:17:53.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.632+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc264045ad0 con 0x7fc2640f5410 2024-01-21T11:17:53.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.632+0000 7fc259ffb700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc25002ff60 0x7fc250032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:53.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.632+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc254041000 con 0x7fc2640f5410 2024-01-21T11:17:53.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.632+0000 7fc2688b2700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc25002ff60 0x7fc250032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:53.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.633+0000 7fc2688b2700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc25002ff60 0x7fc250032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc2600096a0 tx=0x7fc260006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:53.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.637+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc25401d7f0 con 0x7fc2640f5410 2024-01-21T11:17:53.842 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.842+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2640fae60 con 0x7fc2640f5410 2024-01-21T11:17:53.842 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.843+0000 7fc259ffb700 1 -- 172.21.15.178:0/4061619259 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc254010230 con 0x7fc2640f5410 2024-01-21T11:17:53.843 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:53.843 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:53.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc25002ff60 msgr2=0x7fc250032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:53.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc25002ff60 0x7fc250032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc2600096a0 tx=0x7fc260006b40).stop 2024-01-21T11:17:53.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 msgr2=0x7fc264109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:53.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc264109430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fc254013510 tx=0x7fc25400b040).stop 2024-01-21T11:17:53.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 shutdown_connections 2024-01-21T11:17:53.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc25002ff60 0x7fc250032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:53.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 --2- 172.21.15.178:0/4061619259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2640f5410 0x7fc264109430 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:53.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 >> 172.21.15.178:0/4061619259 conn(0x7fc2640f0b70 msgr2=0x7fc2640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:53.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 shutdown_connections 2024-01-21T11:17:53.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:53.844+0000 7fc26b317700 1 -- 172.21.15.178:0/4061619259 wait complete. 2024-01-21T11:17:53.845 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:55.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:53.681635+0000 mgr.smithi089.ufybqw (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:55.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:53.843366+0000 mon.smithi089 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.178:0/4061619259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:55.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:55.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 -- 172.21.15.178:0/2061269243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 msgr2=0x7efe280fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 --2- 172.21.15.178:0/2061269243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe280fb4f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7efe18009fb0 tx=0x7efe18009b30).stop 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 -- 172.21.15.178:0/2061269243 shutdown_connections 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 --2- 172.21.15.178:0/2061269243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe280fb4f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 -- 172.21.15.178:0/2061269243 >> 172.21.15.178:0/2061269243 conn(0x7efe280f0b70 msgr2=0x7efe280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 -- 172.21.15.178:0/2061269243 shutdown_connections 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.429+0000 7efe2e567700 1 -- 172.21.15.178:0/2061269243 wait complete. 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe2e567700 1 Processor -- start 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe2e567700 1 -- start start 2024-01-21T11:17:56.432 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe2e567700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe28108880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe2e567700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28108d70 con 0x7efe28060280 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe27fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe28108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe27fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe28108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42354/0 (socket says 172.21.15.178:42354) 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe27fff700 1 -- 172.21.15.178:0/4244537072 learned_addr learned my addr 172.21.15.178:0/4244537072 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.430+0000 7efe27fff700 1 -- 172.21.15.178:0/4244537072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18005040 con 0x7efe28060280 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.431+0000 7efe27fff700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe28108880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efe18013510 tx=0x7efe1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.431+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe18009320 con 0x7efe28060280 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.431+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe28108f70 con 0x7efe28060280 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.431+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe18013f00 con 0x7efe28060280 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.431+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe18014dc0 con 0x7efe28060280 2024-01-21T11:17:56.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.431+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe28109400 con 0x7efe28060280 2024-01-21T11:17:56.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.432+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe180095f0 con 0x7efe28060280 2024-01-21T11:17:56.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.432+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe28045ad0 con 0x7efe28060280 2024-01-21T11:17:56.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.432+0000 7efe24ff9700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efe1402ff60 0x7efe14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:56.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.432+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe18041020 con 0x7efe28060280 2024-01-21T11:17:56.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.432+0000 7efe277fe700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efe1402ff60 0x7efe14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:56.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.432+0000 7efe277fe700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efe1402ff60 0x7efe14032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efe100096a0 tx=0x7efe10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:56.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.437+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe18014420 con 0x7efe28060280 2024-01-21T11:17:56.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.642+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe2810a250 con 0x7efe28060280 2024-01-21T11:17:56.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.643+0000 7efe24ff9700 1 -- 172.21.15.178:0/4244537072 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe1801c920 con 0x7efe28060280 2024-01-21T11:17:56.644 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:56.644 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:56.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efe1402ff60 msgr2=0x7efe14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:56.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efe1402ff60 0x7efe14032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efe100096a0 tx=0x7efe10006b40).stop 2024-01-21T11:17:56.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 msgr2=0x7efe28108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:56.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe28108880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efe18013510 tx=0x7efe1800b040).stop 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 shutdown_connections 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efe1402ff60 0x7efe14032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 --2- 172.21.15.178:0/4244537072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28060280 0x7efe28108880 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 >> 172.21.15.178:0/4244537072 conn(0x7efe280f0b70 msgr2=0x7efe280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 shutdown_connections 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:56.644+0000 7efe2e567700 1 -- 172.21.15.178:0/4244537072 wait complete. 2024-01-21T11:17:56.646 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:17:57.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:55.681897+0000 mgr.smithi089.ufybqw (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:57.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:56.644555+0000 mon.smithi089 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.178:0/4244537072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:17:58.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:17:58.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:17:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 -- 172.21.15.178:0/444377397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 msgr2=0x7f9c640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 --2- 172.21.15.178:0/444377397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c640f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f9c54009fb0 tx=0x7f9c54009b30).stop 2024-01-21T11:17:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 -- 172.21.15.178:0/444377397 shutdown_connections 2024-01-21T11:17:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 --2- 172.21.15.178:0/444377397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c640f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:59.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 -- 172.21.15.178:0/444377397 >> 172.21.15.178:0/444377397 conn(0x7f9c640f1c70 msgr2=0x7f9c640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:59.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 -- 172.21.15.178:0/444377397 shutdown_connections 2024-01-21T11:17:59.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.133+0000 7f9c6a611700 1 -- 172.21.15.178:0/444377397 wait complete. 2024-01-21T11:17:59.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c6a611700 1 Processor -- start 2024-01-21T11:17:59.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c6a611700 1 -- start start 2024-01-21T11:17:59.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c6a611700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:59.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c6a611700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64106c20 con 0x7f9c640f6510 2024-01-21T11:17:59.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c63fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:59.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c63fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42384/0 (socket says 172.21.15.178:42384) 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.134+0000 7f9c63fff700 1 -- 172.21.15.178:0/1223634183 learned_addr learned my addr 172.21.15.178:0/1223634183 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c63fff700 1 -- 172.21.15.178:0/1223634183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54005040 con 0x7f9c640f6510 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c63fff700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c64106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9c54013510 tx=0x7f9c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c54009320 con 0x7f9c640f6510 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c64106e20 con 0x7f9c640f6510 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c54013f00 con 0x7f9c640f6510 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c54014d90 con 0x7f9c640f6510 2024-01-21T11:17:59.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.135+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c641072b0 con 0x7f9c640f6510 2024-01-21T11:17:59.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.136+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c540095f0 con 0x7f9c640f6510 2024-01-21T11:17:59.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.136+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c64045ad0 con 0x7f9c640f6510 2024-01-21T11:17:59.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.136+0000 7f9c60ff9700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c5002ff60 0x7f9c50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:17:59.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.136+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c54041000 con 0x7f9c640f6510 2024-01-21T11:17:59.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.136+0000 7f9c637fe700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c5002ff60 0x7f9c50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:17:59.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.137+0000 7f9c637fe700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c5002ff60 0x7f9c50032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9c4c0096a0 tx=0x7f9c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:17:59.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.141+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c54014420 con 0x7f9c640f6510 2024-01-21T11:17:59.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:57.682220+0000 mgr.smithi089.ufybqw (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:17:59.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.346+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c640fbf60 con 0x7f9c640f6510 2024-01-21T11:17:59.347 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.347+0000 7f9c60ff9700 1 -- 172.21.15.178:0/1223634183 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c5401d7f0 con 0x7f9c640f6510 2024-01-21T11:17:59.348 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:17:59.348 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:17:59.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c5002ff60 msgr2=0x7f9c50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:59.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c5002ff60 0x7f9c50032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9c4c0096a0 tx=0x7f9c4c006b40).stop 2024-01-21T11:17:59.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 msgr2=0x7f9c64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:17:59.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c64106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9c54013510 tx=0x7f9c5400b040).stop 2024-01-21T11:17:59.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 shutdown_connections 2024-01-21T11:17:59.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c5002ff60 0x7f9c50032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:59.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 --2- 172.21.15.178:0/1223634183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640f6510 0x7f9c64106730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:17:59.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 >> 172.21.15.178:0/1223634183 conn(0x7f9c640f1c70 msgr2=0x7f9c640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:17:59.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 shutdown_connections 2024-01-21T11:17:59.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:17:59.348+0000 7f9c6a611700 1 -- 172.21.15.178:0/1223634183 wait complete. 2024-01-21T11:17:59.350 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:00.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:17:59 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:17:59.348431+0000 mon.smithi089 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.178:0/1223634183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:00.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:00.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:01.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:17:59.682598+0000 mgr.smithi089.ufybqw (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.933+0000 7f96d9d96700 1 -- 172.21.15.178:0/2044943052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 msgr2=0x7f96d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.933+0000 7f96d9d96700 1 --2- 172.21.15.178:0/2044943052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d40f55c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f96c8009fb0 tx=0x7f96c8009b30).stop 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.933+0000 7f96d9d96700 1 -- 172.21.15.178:0/2044943052 shutdown_connections 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.933+0000 7f96d9d96700 1 --2- 172.21.15.178:0/2044943052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d40f55c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.933+0000 7f96d9d96700 1 -- 172.21.15.178:0/2044943052 >> 172.21.15.178:0/2044943052 conn(0x7f96d40f09a0 msgr2=0x7f96d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.934+0000 7f96d9d96700 1 -- 172.21.15.178:0/2044943052 shutdown_connections 2024-01-21T11:18:01.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.934+0000 7f96d9d96700 1 -- 172.21.15.178:0/2044943052 wait complete. 2024-01-21T11:18:01.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.934+0000 7f96d9d96700 1 Processor -- start 2024-01-21T11:18:01.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.934+0000 7f96d9d96700 1 -- start start 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d9d96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d9d96700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41058a0 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d8d94700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d8d94700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42400/0 (socket says 172.21.15.178:42400) 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d8d94700 1 -- 172.21.15.178:0/3757837081 learned_addr learned my addr 172.21.15.178:0/3757837081 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d8d94700 1 -- 172.21.15.178:0/3757837081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c8005040 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d8d94700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d41053b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f96c8013510 tx=0x7f96c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c8009320 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96c8013f00 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d4105aa0 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c8014dc0 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.935+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d4105f30 con 0x7f96d40f51a0 2024-01-21T11:18:01.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.936+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96c801d3d0 con 0x7f96d40f51a0 2024-01-21T11:18:01.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.936+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d4045ad0 con 0x7f96d40f51a0 2024-01-21T11:18:01.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.936+0000 7f96d17fa700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f96c402ff60 0x7f96c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:01.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.936+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96c80412e0 con 0x7f96d40f51a0 2024-01-21T11:18:01.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.937+0000 7f96d3fff700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f96c402ff60 0x7f96c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:01.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.937+0000 7f96d3fff700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f96c402ff60 0x7f96c4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f96c00096a0 tx=0x7f96c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:01.941 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:01.941+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96c801cb10 con 0x7f96d40f51a0 2024-01-21T11:18:02.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.153+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96d4106de0 con 0x7f96d40f51a0 2024-01-21T11:18:02.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.154+0000 7f96d17fa700 1 -- 172.21.15.178:0/3757837081 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f96c801cb10 con 0x7f96d40f51a0 2024-01-21T11:18:02.155 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:02.155 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:02.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f96c402ff60 msgr2=0x7f96c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f96c402ff60 0x7f96c4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f96c00096a0 tx=0x7f96c0006b40).stop 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 msgr2=0x7f96d41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d41053b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f96c8013510 tx=0x7f96c800b040).stop 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 shutdown_connections 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f96c402ff60 0x7f96c4032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 --2- 172.21.15.178:0/3757837081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d40f51a0 0x7f96d41053b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 >> 172.21.15.178:0/3757837081 conn(0x7f96d40f09a0 msgr2=0x7f96d40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 shutdown_connections 2024-01-21T11:18:02.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:02.156+0000 7f96d9d96700 1 -- 172.21.15.178:0/3757837081 wait complete. 2024-01-21T11:18:02.158 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:01.682919+0000 mgr.smithi089.ufybqw (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:03.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:02.155789+0000 mon.smithi089 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.178:0/3757837081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:03.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:03.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 -- 172.21.15.178:0/200698509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 msgr2=0x7fb0940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 --2- 172.21.15.178:0/200698509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb0940f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fb088009fb0 tx=0x7fb088009b30).stop 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 -- 172.21.15.178:0/200698509 shutdown_connections 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 --2- 172.21.15.178:0/200698509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb0940f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 -- 172.21.15.178:0/200698509 >> 172.21.15.178:0/200698509 conn(0x7fb0940f1c70 msgr2=0x7fb0940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 -- 172.21.15.178:0/200698509 shutdown_connections 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.632+0000 7fb098a57700 1 -- 172.21.15.178:0/200698509 wait complete. 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb098a57700 1 Processor -- start 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb098a57700 1 -- start start 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb098a57700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:04.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb098a57700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094106c20 con 0x7fb0940f6510 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb09259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb09259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41006/0 (socket says 172.21.15.178:41006) 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.633+0000 7fb09259c700 1 -- 172.21.15.178:0/3772183616 learned_addr learned my addr 172.21.15.178:0/3772183616 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb09259c700 1 -- 172.21.15.178:0/3772183616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088005040 con 0x7fb0940f6510 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb09259c700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb094106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb088014000 tx=0x7fb08800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0880095f0 con 0x7fb0940f6510 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094106e20 con 0x7fb0940f6510 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb088007e70 con 0x7fb0940f6510 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb088014ea0 con 0x7fb0940f6510 2024-01-21T11:18:04.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.634+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0941072b0 con 0x7fb0940f6510 2024-01-21T11:18:04.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.635+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb088025070 con 0x7fb0940f6510 2024-01-21T11:18:04.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.635+0000 7fb082ffd700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb07c02ff60 0x7fb07c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:04.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.635+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb088040930 con 0x7fb0940f6510 2024-01-21T11:18:04.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.635+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094045ad0 con 0x7fb0940f6510 2024-01-21T11:18:04.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.635+0000 7fb091d9b700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb07c02ff60 0x7fb07c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:04.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.635+0000 7fb091d9b700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb07c02ff60 0x7fb07c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb0840096a0 tx=0x7fb084006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:04.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.639+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb08800f4e0 con 0x7fb0940f6510 2024-01-21T11:18:04.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.847+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0940fbf60 con 0x7fb0940f6510 2024-01-21T11:18:04.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.848+0000 7fb082ffd700 1 -- 172.21.15.178:0/3772183616 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb08802d090 con 0x7fb0940f6510 2024-01-21T11:18:04.849 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:04.849 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb07c02ff60 msgr2=0x7fb07c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb07c02ff60 0x7fb07c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb0840096a0 tx=0x7fb084006b40).stop 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 msgr2=0x7fb094106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb094106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb088014000 tx=0x7fb08800b040).stop 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 shutdown_connections 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb07c02ff60 0x7fb07c032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 --2- 172.21.15.178:0/3772183616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 >> 172.21.15.178:0/3772183616 conn(0x7fb0940f1c70 msgr2=0x7fb0940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 shutdown_connections 2024-01-21T11:18:04.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:04.850+0000 7fb098a57700 1 -- 172.21.15.178:0/3772183616 wait complete. 2024-01-21T11:18:04.852 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:05.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:03.683202+0000 mgr.smithi089.ufybqw (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:06.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:04.849858+0000 mon.smithi089 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.178:0/3772183616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:06.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:06.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:07.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:05.683563+0000 mgr.smithi089.ufybqw (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.344+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/173602510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 msgr2=0x7fdfc80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 --2- 172.21.15.178:0/173602510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc80f55c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fdfb8009fb0 tx=0x7fdfb8009b30).stop 2024-01-21T11:18:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/173602510 shutdown_connections 2024-01-21T11:18:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 --2- 172.21.15.178:0/173602510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc80f55c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/173602510 >> 172.21.15.178:0/173602510 conn(0x7fdfc80f09a0 msgr2=0x7fdfc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/173602510 shutdown_connections 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/173602510 wait complete. 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 Processor -- start 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 -- start start 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.345+0000 7fdfcd6dc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc8060220 con 0x7fdfc80f51a0 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41014/0 (socket says 172.21.15.178:41014) 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc7fff700 1 -- 172.21.15.178:0/4179783270 learned_addr learned my addr 172.21.15.178:0/4179783270 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc7fff700 1 -- 172.21.15.178:0/4179783270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb8005040 con 0x7fdfc80f51a0 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc7fff700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc81091d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fdfb8009830 tx=0x7fdfb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfb8009320 con 0x7fdfc80f51a0 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc8060420 con 0x7fdfc80f51a0 2024-01-21T11:18:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfb80136c0 con 0x7fdfc80f51a0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfb801ee90 con 0x7fdfc80f51a0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.346+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc8109880 con 0x7fdfc80f51a0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.347+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfb801e420 con 0x7fdfc80f51a0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.347+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc8059100 con 0x7fdfc80f51a0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.347+0000 7fdfc4ff9700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfb002ff10 0x7fdfb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.347+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfb8040fa0 con 0x7fdfc80f51a0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.347+0000 7fdfc77fe700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfb002ff10 0x7fdfb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:07.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.348+0000 7fdfc77fe700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfb002ff10 0x7fdfb00323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fdfbc0096a0 tx=0x7fdfbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:07.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.352+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfb80095f0 con 0x7fdfc80f51a0 2024-01-21T11:18:07.559 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.558+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfc80fabe0 con 0x7fdfc80f51a0 2024-01-21T11:18:07.559 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.559+0000 7fdfc4ff9700 1 -- 172.21.15.178:0/4179783270 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfb802c2d0 con 0x7fdfc80f51a0 2024-01-21T11:18:07.560 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:07.560 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:07.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfb002ff10 msgr2=0x7fdfb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:07.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfb002ff10 0x7fdfb00323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fdfbc0096a0 tx=0x7fdfbc006b40).stop 2024-01-21T11:18:07.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 msgr2=0x7fdfc81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:07.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc81091d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fdfb8009830 tx=0x7fdfb800b040).stop 2024-01-21T11:18:07.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 shutdown_connections 2024-01-21T11:18:07.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfb002ff10 0x7fdfb00323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:07.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 --2- 172.21.15.178:0/4179783270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfc80f51a0 0x7fdfc81091d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:07.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 >> 172.21.15.178:0/4179783270 conn(0x7fdfc80f09a0 msgr2=0x7fdfc80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:07.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 shutdown_connections 2024-01-21T11:18:07.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:07.560+0000 7fdfcd6dc700 1 -- 172.21.15.178:0/4179783270 wait complete. 2024-01-21T11:18:07.562 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:08.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:07 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:07.560368+0000 mon.smithi089 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.178:0/4179783270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:08.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:08.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:09.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:07.683843+0000 mgr.smithi089.ufybqw (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:10.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.092+0000 7fe91852c700 1 -- 172.21.15.178:0/735469871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 msgr2=0x7fe9100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:10.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.092+0000 7fe91852c700 1 --2- 172.21.15.178:0/735469871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9100f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fe900009fb0 tx=0x7fe900009b30).stop 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 -- 172.21.15.178:0/735469871 shutdown_connections 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 --2- 172.21.15.178:0/735469871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9100f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 -- 172.21.15.178:0/735469871 >> 172.21.15.178:0/735469871 conn(0x7fe9100f0b70 msgr2=0x7fe9100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 -- 172.21.15.178:0/735469871 shutdown_connections 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 -- 172.21.15.178:0/735469871 wait complete. 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 Processor -- start 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 -- start start 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9101066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe91852c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe910106bc0 con 0x7fe9100f5410 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe9162c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9101066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.093+0000 7fe9162c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9101066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41028/0 (socket says 172.21.15.178:41028) 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe9162c8700 1 -- 172.21.15.178:0/2857192389 learned_addr learned my addr 172.21.15.178:0/2857192389 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:10.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe9162c8700 1 -- 172.21.15.178:0/2857192389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900005040 con 0x7fe9100f5410 2024-01-21T11:18:10.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe9162c8700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9101066d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fe900013510 tx=0x7fe90000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe900009320 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe910106dc0 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe900013f00 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe900014d90 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.094+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe910107250 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.095+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9000095f0 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.095+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe910045ad0 con 0x7fe9100f5410 2024-01-21T11:18:10.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.095+0000 7fe906ffd700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8fc02ff60 0x7fe8fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:10.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.095+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe900041000 con 0x7fe9100f5410 2024-01-21T11:18:10.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.095+0000 7fe915ac7700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8fc02ff60 0x7fe8fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:10.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.096+0000 7fe915ac7700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8fc02ff60 0x7fe8fc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe90c0096a0 tx=0x7fe90c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:10.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.100+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe900014420 con 0x7fe9100f5410 2024-01-21T11:18:10.307 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.307+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe910108100 con 0x7fe9100f5410 2024-01-21T11:18:10.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.307+0000 7fe906ffd700 1 -- 172.21.15.178:0/2857192389 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe900010230 con 0x7fe9100f5410 2024-01-21T11:18:10.308 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:10.308 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8fc02ff60 msgr2=0x7fe8fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8fc02ff60 0x7fe8fc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe90c0096a0 tx=0x7fe90c006b40).stop 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 msgr2=0x7fe9101066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9101066d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fe900013510 tx=0x7fe90000b040).stop 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 shutdown_connections 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe8fc02ff60 0x7fe8fc032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 --2- 172.21.15.178:0/2857192389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9100f5410 0x7fe9101066d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 >> 172.21.15.178:0/2857192389 conn(0x7fe9100f0b70 msgr2=0x7fe9100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 shutdown_connections 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:10.309+0000 7fe91852c700 1 -- 172.21.15.178:0/2857192389 wait complete. 2024-01-21T11:18:10.310 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:11.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:09.684151+0000 mgr.smithi089.ufybqw (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:11.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:10.309019+0000 mon.smithi089 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.178:0/2857192389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:11.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:11.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 -- 172.21.15.178:0/3243731454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 msgr2=0x7f6c800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 --2- 172.21.15.178:0/3243731454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c800f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f6c70009fb0 tx=0x7f6c70009b30).stop 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 -- 172.21.15.178:0/3243731454 shutdown_connections 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 --2- 172.21.15.178:0/3243731454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c800f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 -- 172.21.15.178:0/3243731454 >> 172.21.15.178:0/3243731454 conn(0x7f6c800f1c70 msgr2=0x7f6c800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 -- 172.21.15.178:0/3243731454 shutdown_connections 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.841+0000 7f6c84e22700 1 -- 172.21.15.178:0/3243731454 wait complete. 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c84e22700 1 Processor -- start 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c84e22700 1 -- start start 2024-01-21T11:18:12.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c84e22700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c80106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c84e22700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80106c20 con 0x7f6c800f6510 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c7e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c80106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c7e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c80106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44024/0 (socket says 172.21.15.178:44024) 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c7e59c700 1 -- 172.21.15.178:0/744252853 learned_addr learned my addr 172.21.15.178:0/744252853 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c7e59c700 1 -- 172.21.15.178:0/744252853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c70005040 con 0x7f6c800f6510 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.842+0000 7f6c7e59c700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c80106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f6c70013510 tx=0x7f6c7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c70009320 con 0x7f6c800f6510 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80106e20 con 0x7f6c800f6510 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c70013f00 con 0x7f6c800f6510 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c70014d90 con 0x7f6c800f6510 2024-01-21T11:18:12.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c801072b0 con 0x7f6c800f6510 2024-01-21T11:18:12.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c700095f0 con 0x7f6c800f6510 2024-01-21T11:18:12.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.843+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c80045ad0 con 0x7f6c800f6510 2024-01-21T11:18:12.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.844+0000 7f6c76ffd700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c6c02ff60 0x7f6c6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:12.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.844+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c70041000 con 0x7f6c800f6510 2024-01-21T11:18:12.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.844+0000 7f6c7dd9b700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c6c02ff60 0x7f6c6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:12.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.844+0000 7f6c7dd9b700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c6c02ff60 0x7f6c6c032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6c680096a0 tx=0x7f6c68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:12.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:12.848+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c70014420 con 0x7f6c800f6510 2024-01-21T11:18:13.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.054+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c800fbf60 con 0x7f6c800f6510 2024-01-21T11:18:13.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c76ffd700 1 -- 172.21.15.178:0/744252853 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c7001d7f0 con 0x7f6c800f6510 2024-01-21T11:18:13.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:13.056 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:13.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c6c02ff60 msgr2=0x7f6c6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:13.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c6c02ff60 0x7f6c6c032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6c680096a0 tx=0x7f6c68006b40).stop 2024-01-21T11:18:13.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 msgr2=0x7f6c80106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:13.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c80106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f6c70013510 tx=0x7f6c7000b040).stop 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 shutdown_connections 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6c6c02ff60 0x7f6c6c032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 --2- 172.21.15.178:0/744252853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c800f6510 0x7f6c80106730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 >> 172.21.15.178:0/744252853 conn(0x7f6c800f1c70 msgr2=0x7f6c800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.056+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 shutdown_connections 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:13.057+0000 7f6c84e22700 1 -- 172.21.15.178:0/744252853 wait complete. 2024-01-21T11:18:13.058 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:13.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:11.684464+0000 mgr.smithi089.ufybqw (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:14.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:13.056546+0000 mon.smithi089 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.178:0/744252853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:14.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:14.438 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:13.684733+0000 mgr.smithi089.ufybqw (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:13.775251+0000 mon.smithi089 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:18:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:13.784377+0000 mon.smithi089 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:18:15.606 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 -- 172.21.15.178:0/1216200647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 msgr2=0x7f1dac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 --2- 172.21.15.178:0/1216200647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac0f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1da0009fb0 tx=0x7f1da0009b30).stop 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 -- 172.21.15.178:0/1216200647 shutdown_connections 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 --2- 172.21.15.178:0/1216200647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac0f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 -- 172.21.15.178:0/1216200647 >> 172.21.15.178:0/1216200647 conn(0x7f1dac0f1c70 msgr2=0x7f1dac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 -- 172.21.15.178:0/1216200647 shutdown_connections 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.604+0000 7f1db1266700 1 -- 172.21.15.178:0/1216200647 wait complete. 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1db1266700 1 Processor -- start 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1db1266700 1 -- start start 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1db1266700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1db1266700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac069e20 con 0x7f1dac0f6510 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:15.607 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44046/0 (socket says 172.21.15.178:44046) 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1daaffd700 1 -- 172.21.15.178:0/1948570155 learned_addr learned my addr 172.21.15.178:0/1948570155 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1daaffd700 1 -- 172.21.15.178:0/1948570155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da0005040 con 0x7f1dac0f6510 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.605+0000 7f1daaffd700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac06b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1da0013510 tx=0x7f1da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.606+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da0009320 con 0x7f1dac0f6510 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.606+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac069fc0 con 0x7f1dac0f6510 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.606+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1da0013f00 con 0x7f1dac0f6510 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.606+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da0014d90 con 0x7f1dac0f6510 2024-01-21T11:18:15.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.606+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac06a450 con 0x7f1dac0f6510 2024-01-21T11:18:15.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.606+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1da00263a0 con 0x7f1dac0f6510 2024-01-21T11:18:15.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.607+0000 7f1d937fe700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1d9402ff60 0x7f1d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:15.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.607+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1da0041030 con 0x7f1dac0f6510 2024-01-21T11:18:15.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.607+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac059100 con 0x7f1dac0f6510 2024-01-21T11:18:15.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.607+0000 7f1daa7fc700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1d9402ff60 0x7f1d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:15.609 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.607+0000 7f1daa7fc700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1d9402ff60 0x7f1d94032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1d9c0096a0 tx=0x7f1d9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:15.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.611+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1da0012920 con 0x7f1dac0f6510 2024-01-21T11:18:15.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.817+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1dac06ab50 con 0x7f1dac0f6510 2024-01-21T11:18:15.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.818+0000 7f1d937fe700 1 -- 172.21.15.178:0/1948570155 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1da002b020 con 0x7f1dac0f6510 2024-01-21T11:18:15.818 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:15.819 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1d9402ff60 msgr2=0x7f1d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1d9402ff60 0x7f1d94032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1d9c0096a0 tx=0x7f1d9c006b40).stop 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 msgr2=0x7f1dac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac06b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1da0013510 tx=0x7f1da000b040).stop 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 shutdown_connections 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1d9402ff60 0x7f1d94032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:15.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 --2- 172.21.15.178:0/1948570155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1dac0f6510 0x7f1dac06b9a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:15.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 >> 172.21.15.178:0/1948570155 conn(0x7f1dac0f1c70 msgr2=0x7f1dac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:15.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 shutdown_connections 2024-01-21T11:18:15.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:15.819+0000 7f1db1266700 1 -- 172.21.15.178:0/1948570155 wait complete. 2024-01-21T11:18:15.821 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:17.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:15.685013+0000 mgr.smithi089.ufybqw (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:17.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:15.819272+0000 mon.smithi089 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.178:0/1948570155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:17.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:17.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:18.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:17.685415+0000 mgr.smithi089.ufybqw (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:18.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 -- 172.21.15.178:0/2475217391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 msgr2=0x7f7b180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:18.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 --2- 172.21.15.178:0/2475217391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f7b08009fb0 tx=0x7f7b08009b30).stop 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 -- 172.21.15.178:0/2475217391 shutdown_connections 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 --2- 172.21.15.178:0/2475217391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 -- 172.21.15.178:0/2475217391 >> 172.21.15.178:0/2475217391 conn(0x7f7b180f0b70 msgr2=0x7f7b180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 -- 172.21.15.178:0/2475217391 shutdown_connections 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 -- 172.21.15.178:0/2475217391 wait complete. 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.329+0000 7f7b1ff14700 1 Processor -- start 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1ff14700 1 -- start start 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1ff14700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:18.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1ff14700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180fb820 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1dcb0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1dcb0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44066/0 (socket says 172.21.15.178:44066) 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1dcb0700 1 -- 172.21.15.178:0/3641429192 learned_addr learned my addr 172.21.15.178:0/3641429192 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1dcb0700 1 -- 172.21.15.178:0/3641429192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08005040 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.330+0000 7f7b1dcb0700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180fd3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7b08013510 tx=0x7f7b0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b08009320 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b08013f00 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b180fb9c0 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b08014d90 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b180fbe50 con 0x7f7b180f5410 2024-01-21T11:18:18.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b080095f0 con 0x7f7b180f5410 2024-01-21T11:18:18.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b18059100 con 0x7f7b180f5410 2024-01-21T11:18:18.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.331+0000 7f7b0e7fc700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7b0402ff60 0x7f7b04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:18.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.332+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b080410c0 con 0x7f7b180f5410 2024-01-21T11:18:18.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.332+0000 7f7b1d4af700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7b0402ff60 0x7f7b04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:18.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.332+0000 7f7b1d4af700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7b0402ff60 0x7f7b04032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7b140096a0 tx=0x7f7b14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:18.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.336+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b08014420 con 0x7f7b180f5410 2024-01-21T11:18:18.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.542+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b180fae60 con 0x7f7b180f5410 2024-01-21T11:18:18.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.543+0000 7f7b0e7fc700 1 -- 172.21.15.178:0/3641429192 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b08010230 con 0x7f7b180f5410 2024-01-21T11:18:18.543 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:18.543 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:18.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7b0402ff60 msgr2=0x7f7b04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7b0402ff60 0x7f7b04032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7b140096a0 tx=0x7f7b14006b40).stop 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 msgr2=0x7f7b180fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180fd3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7b08013510 tx=0x7f7b0800b040).stop 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 shutdown_connections 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f7b0402ff60 0x7f7b04032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 --2- 172.21.15.178:0/3641429192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b180f5410 0x7f7b180fd3a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 >> 172.21.15.178:0/3641429192 conn(0x7f7b180f0b70 msgr2=0x7f7b1805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 shutdown_connections 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:18.545+0000 7f7b1ff14700 1 -- 172.21.15.178:0/3641429192 wait complete. 2024-01-21T11:18:18.545 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:19.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:18.543726+0000 mon.smithi089 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.178:0/3641429192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:19.958 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:19.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.083+0000 7f1f0b449700 1 -- 172.21.15.178:0/979556701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 msgr2=0x7f1f040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.083+0000 7f1f0b449700 1 --2- 172.21.15.178:0/979556701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f040f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f1ef4009fb0 tx=0x7f1ef4009b30).stop 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.083+0000 7f1f0b449700 1 -- 172.21.15.178:0/979556701 shutdown_connections 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.083+0000 7f1f0b449700 1 --2- 172.21.15.178:0/979556701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f040f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.083+0000 7f1f0b449700 1 -- 172.21.15.178:0/979556701 >> 172.21.15.178:0/979556701 conn(0x7f1f040f0b70 msgr2=0x7f1f040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.084+0000 7f1f0b449700 1 -- 172.21.15.178:0/979556701 shutdown_connections 2024-01-21T11:18:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.084+0000 7f1f0b449700 1 -- 172.21.15.178:0/979556701 wait complete. 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.084+0000 7f1f0b449700 1 Processor -- start 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.084+0000 7f1f0b449700 1 -- start start 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.084+0000 7f1f0b449700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.084+0000 7f1f0b449700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f04106c20 con 0x7f1f040f5410 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f091e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f091e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44078/0 (socket says 172.21.15.178:44078) 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f091e5700 1 -- 172.21.15.178:0/1909933771 learned_addr learned my addr 172.21.15.178:0/1909933771 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f091e5700 1 -- 172.21.15.178:0/1909933771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef4005040 con 0x7f1f040f5410 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f091e5700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f1ef4014000 tx=0x7f1ef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ef40095f0 con 0x7f1f040f5410 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f04106e20 con 0x7f1f040f5410 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ef4007e70 con 0x7f1f040f5410 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ef4014ea0 con 0x7f1f040f5410 2024-01-21T11:18:21.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.085+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f041072b0 con 0x7f1f040f5410 2024-01-21T11:18:21.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.086+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ef4025070 con 0x7f1f040f5410 2024-01-21T11:18:21.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.086+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04045ad0 con 0x7f1f040f5410 2024-01-21T11:18:21.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.086+0000 7f1ef9ffb700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ffb0 0x7f1ef0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:21.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.086+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ef40409e0 con 0x7f1f040f5410 2024-01-21T11:18:21.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.087+0000 7f1f089e4700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ffb0 0x7f1ef0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:21.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.087+0000 7f1f089e4700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ffb0 0x7f1ef0032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1f000096a0 tx=0x7f1f00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:21.090 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.091+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ef40079c0 con 0x7f1f040f5410 2024-01-21T11:18:21.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:19.685800+0000 mgr.smithi089.ufybqw (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:21.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.297+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f040fbf90 con 0x7f1f040f5410 2024-01-21T11:18:21.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.297+0000 7f1ef9ffb700 1 -- 172.21.15.178:0/1909933771 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ef400f300 con 0x7f1f040f5410 2024-01-21T11:18:21.297 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:21.297 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ffb0 msgr2=0x7f1ef0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ffb0 0x7f1ef0032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1f000096a0 tx=0x7f1f00006b40).stop 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 msgr2=0x7f1f04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f1ef4014000 tx=0x7f1ef400b040).stop 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 shutdown_connections 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1ef002ffb0 0x7f1ef0032470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 --2- 172.21.15.178:0/1909933771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f040f5410 0x7f1f04106730 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 >> 172.21.15.178:0/1909933771 conn(0x7f1f040f0b70 msgr2=0x7f1f040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:21.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 shutdown_connections 2024-01-21T11:18:21.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:21.299+0000 7f1f0b449700 1 -- 172.21.15.178:0/1909933771 wait complete. 2024-01-21T11:18:21.300 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:22.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:21.298179+0000 mon.smithi089 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.178:0/1909933771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:22.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:22.683 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:23.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:21.686132+0000 mgr.smithi089.ufybqw (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.831+0000 7f32f2975700 1 -- 172.21.15.178:0/2760023657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 msgr2=0x7f32ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.831+0000 7f32f2975700 1 --2- 172.21.15.178:0/2760023657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f32dc009fb0 tx=0x7f32dc009b30).stop 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 -- 172.21.15.178:0/2760023657 shutdown_connections 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 --2- 172.21.15.178:0/2760023657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 -- 172.21.15.178:0/2760023657 >> 172.21.15.178:0/2760023657 conn(0x7f32ec0f0b70 msgr2=0x7f32ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 -- 172.21.15.178:0/2760023657 shutdown_connections 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 -- 172.21.15.178:0/2760023657 wait complete. 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 Processor -- start 2024-01-21T11:18:23.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 -- start start 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.832+0000 7f32f2975700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec105ac0 con 0x7f32ec0f5410 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32ebfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32ebfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40900/0 (socket says 172.21.15.178:40900) 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32ebfff700 1 -- 172.21.15.178:0/696131786 learned_addr learned my addr 172.21.15.178:0/696131786 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32ebfff700 1 -- 172.21.15.178:0/696131786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc005040 con 0x7f32ec0f5410 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32ebfff700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec1055d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f32dc013510 tx=0x7f32dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32dc009320 con 0x7f32ec0f5410 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ec105cc0 con 0x7f32ec0f5410 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32dc013f00 con 0x7f32ec0f5410 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32dc014d90 con 0x7f32ec0f5410 2024-01-21T11:18:23.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.833+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec106150 con 0x7f32ec0f5410 2024-01-21T11:18:23.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.834+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32dc0095f0 con 0x7f32ec0f5410 2024-01-21T11:18:23.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.834+0000 7f32e8ff9700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32d402ff60 0x7f32d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:23.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.834+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec045ad0 con 0x7f32ec0f5410 2024-01-21T11:18:23.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.834+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32dc041000 con 0x7f32ec0f5410 2024-01-21T11:18:23.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.835+0000 7f32eb7fe700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32d402ff60 0x7f32d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:23.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.835+0000 7f32eb7fe700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32d402ff60 0x7f32d4032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f32e00096a0 tx=0x7f32e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:23.839 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:23.839+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32dc014420 con 0x7f32ec0f5410 2024-01-21T11:18:24.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.048+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32ec107000 con 0x7f32ec0f5410 2024-01-21T11:18:24.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.048+0000 7f32e8ff9700 1 -- 172.21.15.178:0/696131786 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32dc02c090 con 0x7f32ec0f5410 2024-01-21T11:18:24.048 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:24.049 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32d402ff60 msgr2=0x7f32d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32d402ff60 0x7f32d4032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f32e00096a0 tx=0x7f32e0006b40).stop 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 msgr2=0x7f32ec1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec1055d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f32dc013510 tx=0x7f32dc00b040).stop 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 shutdown_connections 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f32d402ff60 0x7f32d4032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 --2- 172.21.15.178:0/696131786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32ec0f5410 0x7f32ec1055d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 >> 172.21.15.178:0/696131786 conn(0x7f32ec0f0b70 msgr2=0x7f32ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 shutdown_connections 2024-01-21T11:18:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:24.050+0000 7f32f2975700 1 -- 172.21.15.178:0/696131786 wait complete. 2024-01-21T11:18:24.051 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:25.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:23.686534+0000 mgr.smithi089.ufybqw (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:25.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:24.049281+0000 mon.smithi089 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.178:0/696131786' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:25.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:25.440 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:26.592 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1220249722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 msgr2=0x7fcda80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 --2- 172.21.15.178:0/1220249722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda80f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fcd98009fb0 tx=0x7fcd98009b30).stop 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1220249722 shutdown_connections 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 --2- 172.21.15.178:0/1220249722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda80f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1220249722 >> 172.21.15.178:0/1220249722 conn(0x7fcda80f1c70 msgr2=0x7fcda80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1220249722 shutdown_connections 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1220249722 wait complete. 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.591+0000 7fcdae1ce700 1 Processor -- start 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcdae1ce700 1 -- start start 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcdae1ce700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcdae1ce700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda8069e20 con 0x7fcda80f6510 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcda77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:26.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcda77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40912/0 (socket says 172.21.15.178:40912) 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcda77fe700 1 -- 172.21.15.178:0/1753587456 learned_addr learned my addr 172.21.15.178:0/1753587456 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.592+0000 7fcda77fe700 1 -- 172.21.15.178:0/1753587456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98005040 con 0x7fcda80f6510 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcda77fe700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda806b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcd98013510 tx=0x7fcd9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd98009320 con 0x7fcda80f6510 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda8069fc0 con 0x7fcda80f6510 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd98013f00 con 0x7fcda80f6510 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd98014d90 con 0x7fcda80f6510 2024-01-21T11:18:26.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda806a450 con 0x7fcda80f6510 2024-01-21T11:18:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.593+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd980095f0 con 0x7fcda80f6510 2024-01-21T11:18:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.594+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda8059100 con 0x7fcda80f6510 2024-01-21T11:18:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.594+0000 7fcd9ffff700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcd9402ff60 0x7fcd94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.594+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd98041000 con 0x7fcda80f6510 2024-01-21T11:18:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.594+0000 7fcda6ffd700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcd9402ff60 0x7fcd94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.594+0000 7fcda6ffd700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcd9402ff60 0x7fcd94032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcd900096a0 tx=0x7fcd90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.598+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd98014420 con 0x7fcda80f6510 2024-01-21T11:18:26.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.805+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcda80fbf60 con 0x7fcda80f6510 2024-01-21T11:18:26.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.806+0000 7fcd9ffff700 1 -- 172.21.15.178:0/1753587456 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd9801d7f0 con 0x7fcda80f6510 2024-01-21T11:18:26.806 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:26.806 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:26.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcd9402ff60 msgr2=0x7fcd94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:26.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcd9402ff60 0x7fcd94032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcd900096a0 tx=0x7fcd90006b40).stop 2024-01-21T11:18:26.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 msgr2=0x7fcda806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:26.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda806b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcd98013510 tx=0x7fcd9800b040).stop 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 shutdown_connections 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fcd9402ff60 0x7fcd94032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 --2- 172.21.15.178:0/1753587456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcda80f6510 0x7fcda806b9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.807+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 >> 172.21.15.178:0/1753587456 conn(0x7fcda80f1c70 msgr2=0x7fcda80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.808+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 shutdown_connections 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:26.808+0000 7fcdae1ce700 1 -- 172.21.15.178:0/1753587456 wait complete. 2024-01-21T11:18:26.808 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:27.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:25.686896+0000 mgr.smithi089.ufybqw (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:28.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:26.806666+0000 mon.smithi089 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.178:0/1753587456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:28.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:28.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:29.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:27.687292+0000 mgr.smithi089.ufybqw (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 -- 172.21.15.178:0/3317932975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 msgr2=0x7f0d6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 --2- 172.21.15.178:0/3317932975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c0f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f0d60009fb0 tx=0x7f0d60009b30).stop 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 -- 172.21.15.178:0/3317932975 shutdown_connections 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 --2- 172.21.15.178:0/3317932975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c0f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 -- 172.21.15.178:0/3317932975 >> 172.21.15.178:0/3317932975 conn(0x7f0d6c0f1c70 msgr2=0x7f0d6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 -- 172.21.15.178:0/3317932975 shutdown_connections 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.369+0000 7f0d73c11700 1 -- 172.21.15.178:0/3317932975 wait complete. 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d73c11700 1 Processor -- start 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d73c11700 1 -- start start 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d73c11700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d73c11700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c106c20 con 0x7f0d6c0f6510 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d719ad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d719ad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40932/0 (socket says 172.21.15.178:40932) 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d719ad700 1 -- 172.21.15.178:0/611477900 learned_addr learned my addr 172.21.15.178:0/611477900 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d719ad700 1 -- 172.21.15.178:0/611477900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d60005040 con 0x7f0d6c0f6510 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.370+0000 7f0d719ad700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0d60013510 tx=0x7f0d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d60009320 con 0x7f0d6c0f6510 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d60013f00 con 0x7f0d6c0f6510 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c106e20 con 0x7f0d6c0f6510 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d60014d90 con 0x7f0d6c0f6510 2024-01-21T11:18:29.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c1072b0 con 0x7f0d6c0f6510 2024-01-21T11:18:29.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d600095f0 con 0x7f0d6c0f6510 2024-01-21T11:18:29.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.371+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c045ad0 con 0x7f0d6c0f6510 2024-01-21T11:18:29.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.372+0000 7f0d5e7fc700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:29.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.372+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d60040fe0 con 0x7f0d6c0f6510 2024-01-21T11:18:29.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.372+0000 7f0d711ac700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:29.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.372+0000 7f0d711ac700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0d5802ff60 0x7f0d58032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:29.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.376+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d6001d7f0 con 0x7f0d6c0f6510 2024-01-21T11:18:29.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.581+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d6c0fbf60 con 0x7f0d6c0f6510 2024-01-21T11:18:29.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.582+0000 7f0d5e7fc700 1 -- 172.21.15.178:0/611477900 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d60010230 con 0x7f0d6c0f6510 2024-01-21T11:18:29.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:29.582 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0d5802ff60 msgr2=0x7f0d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0d5802ff60 0x7f0d58032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).stop 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 msgr2=0x7f0d6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0d60013510 tx=0x7f0d6000b040).stop 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 shutdown_connections 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 --2- 172.21.15.178:0/611477900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d6c0f6510 0x7f0d6c106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 >> 172.21.15.178:0/611477900 conn(0x7f0d6c0f1c70 msgr2=0x7f0d6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 shutdown_connections 2024-01-21T11:18:29.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:29.584+0000 7f0d73c11700 1 -- 172.21.15.178:0/611477900 wait complete. 2024-01-21T11:18:29.585 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:30.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:29.582855+0000 mon.smithi089 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.178:0/611477900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:30.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:30.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:31.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:29.687633+0000 mgr.smithi089.ufybqw (mgr.14162) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 -- 172.21.15.178:0/833847404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 msgr2=0x7f2e2c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 --2- 172.21.15.178:0/833847404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f4b60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f2e20009fb0 tx=0x7f2e20009b30).stop 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 -- 172.21.15.178:0/833847404 shutdown_connections 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 --2- 172.21.15.178:0/833847404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f4b60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 -- 172.21.15.178:0/833847404 >> 172.21.15.178:0/833847404 conn(0x7f2e2c0f0260 msgr2=0x7f2e2c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 -- 172.21.15.178:0/833847404 shutdown_connections 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.107+0000 7f2e331d2700 1 -- 172.21.15.178:0/833847404 wait complete. 2024-01-21T11:18:32.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e331d2700 1 Processor -- start 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e331d2700 1 -- start start 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e331d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e331d2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0f5770 con 0x7f2e2c0f4740 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e30f6e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e30f6e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40942/0 (socket says 172.21.15.178:40942) 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.108+0000 7f2e30f6e700 1 -- 172.21.15.178:0/3351071222 learned_addr learned my addr 172.21.15.178:0/3351071222 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e30f6e700 1 -- 172.21.15.178:0/3351071222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e20005040 con 0x7f2e2c0f4740 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e30f6e700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f72f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2e20013510 tx=0x7f2e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e20009320 con 0x7f2e2c0f4740 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c0f5910 con 0x7f2e2c0f4740 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e20013f00 con 0x7f2e2c0f4740 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e20014d90 con 0x7f2e2c0f4740 2024-01-21T11:18:32.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.109+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c0f5da0 con 0x7f2e2c0f4740 2024-01-21T11:18:32.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.110+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e200095f0 con 0x7f2e2c0f4740 2024-01-21T11:18:32.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.110+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c059100 con 0x7f2e2c0f4740 2024-01-21T11:18:32.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.110+0000 7f2e297fa700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e1c02ff60 0x7f2e1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:32.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.110+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e20041000 con 0x7f2e2c0f4740 2024-01-21T11:18:32.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.110+0000 7f2e2bfff700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e1c02ff60 0x7f2e1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:32.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.111+0000 7f2e2bfff700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e1c02ff60 0x7f2e1c032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f2e180096a0 tx=0x7f2e18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:32.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.115+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e20014420 con 0x7f2e2c0f4740 2024-01-21T11:18:32.322 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.322+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e2c0fbf40 con 0x7f2e2c0f4740 2024-01-21T11:18:32.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.323+0000 7f2e297fa700 1 -- 172.21.15.178:0/3351071222 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e20010230 con 0x7f2e2c0f4740 2024-01-21T11:18:32.323 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:32.323 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:32.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.324+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e1c02ff60 msgr2=0x7f2e1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.324+0000 7f2e331d2700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e1c02ff60 0x7f2e1c032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f2e180096a0 tx=0x7f2e18006b40).stop 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.324+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 msgr2=0x7f2e2c0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.324+0000 7f2e331d2700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f72f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2e20013510 tx=0x7f2e2000b040).stop 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.325+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 shutdown_connections 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.325+0000 7f2e331d2700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2e1c02ff60 0x7f2e1c032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.325+0000 7f2e331d2700 1 --2- 172.21.15.178:0/3351071222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e2c0f4740 0x7f2e2c0f72f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.325+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 >> 172.21.15.178:0/3351071222 conn(0x7f2e2c0f0260 msgr2=0x7f2e2c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.325+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 shutdown_connections 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:32.325+0000 7f2e331d2700 1 -- 172.21.15.178:0/3351071222 wait complete. 2024-01-21T11:18:32.325 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:33.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:31.687957+0000 mgr.smithi089.ufybqw (mgr.14162) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:33.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:32.323735+0000 mon.smithi089 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.178:0/3351071222' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:33.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:33.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:34.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 -- 172.21.15.178:0/4035295698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 msgr2=0x7f1e880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 --2- 172.21.15.178:0/4035295698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e880f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f1e70009fb0 tx=0x7f1e70009b30).stop 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 -- 172.21.15.178:0/4035295698 shutdown_connections 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 --2- 172.21.15.178:0/4035295698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e880f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 -- 172.21.15.178:0/4035295698 >> 172.21.15.178:0/4035295698 conn(0x7f1e880f0b70 msgr2=0x7f1e880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 -- 172.21.15.178:0/4035295698 shutdown_connections 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.853+0000 7f1e8d597700 1 -- 172.21.15.178:0/4035295698 wait complete. 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e8d597700 1 Processor -- start 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e8d597700 1 -- start start 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e8d597700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e8810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e8d597700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e88107210 con 0x7f1e880f5410 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e86ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e8810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e86ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e8810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52382/0 (socket says 172.21.15.178:52382) 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.854+0000 7f1e86ffd700 1 -- 172.21.15.178:0/2136056720 learned_addr learned my addr 172.21.15.178:0/2136056720 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e86ffd700 1 -- 172.21.15.178:0/2136056720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e70005040 con 0x7f1e880f5410 2024-01-21T11:18:34.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e86ffd700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e8810b820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1e70013510 tx=0x7f1e7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:34.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e70009320 con 0x7f1e880f5410 2024-01-21T11:18:34.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e88060220 con 0x7f1e880f5410 2024-01-21T11:18:34.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e70013f00 con 0x7f1e880f5410 2024-01-21T11:18:34.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e70014d90 con 0x7f1e880f5410 2024-01-21T11:18:34.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.855+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8810bd10 con 0x7f1e880f5410 2024-01-21T11:18:34.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.856+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e70014420 con 0x7f1e880f5410 2024-01-21T11:18:34.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.856+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e88045ad0 con 0x7f1e880f5410 2024-01-21T11:18:34.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.856+0000 7f1e7f7fe700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1e7402ff60 0x7f1e74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:34.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.856+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e70041000 con 0x7f1e880f5410 2024-01-21T11:18:34.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.856+0000 7f1e867fc700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1e7402ff60 0x7f1e74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:34.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.856+0000 7f1e867fc700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1e7402ff60 0x7f1e74032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1e780096a0 tx=0x7f1e78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:34.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:34.861+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e700095f0 con 0x7f1e880f5410 2024-01-21T11:18:35.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.067+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e880fae60 con 0x7f1e880f5410 2024-01-21T11:18:35.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.067+0000 7f1e7f7fe700 1 -- 172.21.15.178:0/2136056720 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e7001db70 con 0x7f1e880f5410 2024-01-21T11:18:35.067 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:35.067 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1e7402ff60 msgr2=0x7f1e74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1e7402ff60 0x7f1e74032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1e780096a0 tx=0x7f1e78006b40).stop 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 msgr2=0x7f1e8810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e8810b820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1e70013510 tx=0x7f1e7000b040).stop 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 shutdown_connections 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1e7402ff60 0x7f1e74032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 --2- 172.21.15.178:0/2136056720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e880f5410 0x7f1e8810b820 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 >> 172.21.15.178:0/2136056720 conn(0x7f1e880f0b70 msgr2=0x7f1e880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 shutdown_connections 2024-01-21T11:18:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:35.069+0000 7f1e8d597700 1 -- 172.21.15.178:0/2136056720 wait complete. 2024-01-21T11:18:35.070 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:35.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:33.688240+0000 mgr.smithi089.ufybqw (mgr.14162) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:36.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:35 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:35.067979+0000 mon.smithi089 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.178:0/2136056720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:36.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:36.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:37.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:35.688540+0000 mgr.smithi089.ufybqw (mgr.14162) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:37.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 -- 172.21.15.178:0/1266941479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 msgr2=0x7f81280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:37.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 --2- 172.21.15.178:0/1266941479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81280f55c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8118009fb0 tx=0x7f8118009b30).stop 2024-01-21T11:18:37.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 -- 172.21.15.178:0/1266941479 shutdown_connections 2024-01-21T11:18:37.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 --2- 172.21.15.178:0/1266941479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81280f55c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 -- 172.21.15.178:0/1266941479 >> 172.21.15.178:0/1266941479 conn(0x7f81280f09a0 msgr2=0x7f81280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 -- 172.21.15.178:0/1266941479 shutdown_connections 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 -- 172.21.15.178:0/1266941479 wait complete. 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 Processor -- start 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 -- start start 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.601+0000 7f812e3b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81281064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812e3b3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81281069b0 con 0x7f81280f51a0 2024-01-21T11:18:37.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812d3b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81281064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:37.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812d3b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81281064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52400/0 (socket says 172.21.15.178:52400) 2024-01-21T11:18:37.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812d3b1700 1 -- 172.21.15.178:0/1854507644 learned_addr learned my addr 172.21.15.178:0/1854507644 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:37.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812d3b1700 1 -- 172.21.15.178:0/1854507644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8118005040 con 0x7f81280f51a0 2024-01-21T11:18:37.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812d3b1700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81281064c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8118013510 tx=0x7f811800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:37.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118009320 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8128106bb0 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8118013f00 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.602+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118014dc0 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.603+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8128107040 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.603+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81180263a0 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.603+0000 7f811dffb700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f811402ffb0 0x7f8114032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.604+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8118040ff0 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.604+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8128045ad0 con 0x7f81280f51a0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.604+0000 7f812cbb0700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f811402ffb0 0x7f8114032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:37.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.604+0000 7f812cbb0700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f811402ffb0 0x7f8114032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f81240096a0 tx=0x7f8124006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:37.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.608+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8118012920 con 0x7f81280f51a0 2024-01-21T11:18:37.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.815+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8128107d80 con 0x7f81280f51a0 2024-01-21T11:18:37.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.815+0000 7f811dffb700 1 -- 172.21.15.178:0/1854507644 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f811802b020 con 0x7f81280f51a0 2024-01-21T11:18:37.816 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:37.816 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f811402ffb0 msgr2=0x7f8114032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f811402ffb0 0x7f8114032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f81240096a0 tx=0x7f8124006b40).stop 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 msgr2=0x7f81281064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81281064c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8118013510 tx=0x7f811800b040).stop 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 shutdown_connections 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f811402ffb0 0x7f8114032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 --2- 172.21.15.178:0/1854507644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81280f51a0 0x7f81281064c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 >> 172.21.15.178:0/1854507644 conn(0x7f81280f09a0 msgr2=0x7f81280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 shutdown_connections 2024-01-21T11:18:37.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:37.817+0000 7f812e3b3700 1 -- 172.21.15.178:0/1854507644 wait complete. 2024-01-21T11:18:37.818 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:39.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:37.688931+0000 mgr.smithi089.ufybqw (mgr.14162) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:39.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:37.816179+0000 mon.smithi089 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.178:0/1854507644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:39.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:39.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.384+0000 7f5e9935f700 1 -- 172.21.15.178:0/270449802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 msgr2=0x7f5e940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.384+0000 7f5e9935f700 1 --2- 172.21.15.178:0/270449802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e940f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f5e84009fb0 tx=0x7f5e84009b30).stop 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.384+0000 7f5e9935f700 1 -- 172.21.15.178:0/270449802 shutdown_connections 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.384+0000 7f5e9935f700 1 --2- 172.21.15.178:0/270449802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e940f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.384+0000 7f5e9935f700 1 -- 172.21.15.178:0/270449802 >> 172.21.15.178:0/270449802 conn(0x7f5e940f09a0 msgr2=0x7f5e940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.385+0000 7f5e9935f700 1 -- 172.21.15.178:0/270449802 shutdown_connections 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.385+0000 7f5e9935f700 1 -- 172.21.15.178:0/270449802 wait complete. 2024-01-21T11:18:40.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.385+0000 7f5e9935f700 1 Processor -- start 2024-01-21T11:18:40.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.385+0000 7f5e9935f700 1 -- start start 2024-01-21T11:18:40.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.385+0000 7f5e9935f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e941064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:40.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.385+0000 7f5e9935f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e941069d0 con 0x7f5e940f51a0 2024-01-21T11:18:40.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e93fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e941064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:40.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e93fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e941064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52416/0 (socket says 172.21.15.178:52416) 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e93fff700 1 -- 172.21.15.178:0/4246316149 learned_addr learned my addr 172.21.15.178:0/4246316149 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e93fff700 1 -- 172.21.15.178:0/4246316149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e84005040 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e93fff700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e941064e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5e84013510 tx=0x7f5e8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e84009320 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e94106bd0 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e84013f00 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e84014dc0 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.386+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e94107060 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.387+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e840095f0 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.387+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e94045ad0 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.387+0000 7f5e90ff9700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7c02ff60 0x7f5e7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.387+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e84041100 con 0x7f5e940f51a0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.387+0000 7f5e937fe700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7c02ff60 0x7f5e7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:40.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.388+0000 7f5e937fe700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7c02ff60 0x7f5e7c032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5e880096a0 tx=0x7f5e88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:40.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.392+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e8401c790 con 0x7f5e940f51a0 2024-01-21T11:18:40.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.601+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e94107f10 con 0x7f5e940f51a0 2024-01-21T11:18:40.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.602+0000 7f5e90ff9700 1 -- 172.21.15.178:0/4246316149 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e84010230 con 0x7f5e940f51a0 2024-01-21T11:18:40.602 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:40.602 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:40.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7c02ff60 msgr2=0x7f5e7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7c02ff60 0x7f5e7c032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5e880096a0 tx=0x7f5e88006b40).stop 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 msgr2=0x7f5e941064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e941064e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5e84013510 tx=0x7f5e8400b040).stop 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 shutdown_connections 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7c02ff60 0x7f5e7c032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 --2- 172.21.15.178:0/4246316149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e940f51a0 0x7f5e941064e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 >> 172.21.15.178:0/4246316149 conn(0x7f5e940f09a0 msgr2=0x7f5e9405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 shutdown_connections 2024-01-21T11:18:40.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:40.603+0000 7f5e9935f700 1 -- 172.21.15.178:0/4246316149 wait complete. 2024-01-21T11:18:40.605 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:41.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:39.689200+0000 mgr.smithi089.ufybqw (mgr.14162) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:41.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:40.602322+0000 mon.smithi089 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.178:0/4246316149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:42.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:42.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 -- 172.21.15.178:0/2816939591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 msgr2=0x7f61d40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 --2- 172.21.15.178:0/2816939591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d40f90f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f61c4009fb0 tx=0x7f61c4009b30).stop 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 -- 172.21.15.178:0/2816939591 shutdown_connections 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 --2- 172.21.15.178:0/2816939591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d40f90f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 -- 172.21.15.178:0/2816939591 >> 172.21.15.178:0/2816939591 conn(0x7f61d40f0b70 msgr2=0x7f61d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 -- 172.21.15.178:0/2816939591 shutdown_connections 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.169+0000 7f61dc743700 1 -- 172.21.15.178:0/2816939591 wait complete. 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61dc743700 1 Processor -- start 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61dc743700 1 -- start start 2024-01-21T11:18:43.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61dc743700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61dc743700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d4106bd0 con 0x7f61d4060280 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61da4df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61da4df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40764/0 (socket says 172.21.15.178:40764) 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61da4df700 1 -- 172.21.15.178:0/1611234385 learned_addr learned my addr 172.21.15.178:0/1611234385 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61da4df700 1 -- 172.21.15.178:0/1611234385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4005040 con 0x7f61d4060280 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.170+0000 7f61da4df700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d41066e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f61c4013510 tx=0x7f61c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.171+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c4009320 con 0x7f61d4060280 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.171+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61c4013f00 con 0x7f61d4060280 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.171+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d4106dd0 con 0x7f61d4060280 2024-01-21T11:18:43.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.171+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c4014dc0 con 0x7f61d4060280 2024-01-21T11:18:43.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.171+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d4107260 con 0x7f61d4060280 2024-01-21T11:18:43.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.171+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61c40095f0 con 0x7f61d4060280 2024-01-21T11:18:43.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.172+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d4045ad0 con 0x7f61d4060280 2024-01-21T11:18:43.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.172+0000 7f61caffd700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:43.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.172+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61c40410b0 con 0x7f61d4060280 2024-01-21T11:18:43.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.172+0000 7f61d9cde700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:43.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.172+0000 7f61d9cde700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f61d00096a0 tx=0x7f61d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:43.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.176+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61c4014420 con 0x7f61d4060280 2024-01-21T11:18:43.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:41.689556+0000 mgr.smithi089.ufybqw (mgr.14162) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:43.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.387+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61d4108110 con 0x7f61d4060280 2024-01-21T11:18:43.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.388+0000 7f61caffd700 1 -- 172.21.15.178:0/1611234385 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61c401d7f0 con 0x7f61d4060280 2024-01-21T11:18:43.388 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:43.388 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 msgr2=0x7f61c0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f61d00096a0 tx=0x7f61d0006b40).stop 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 msgr2=0x7f61d41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d41066e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f61c4013510 tx=0x7f61c400b040).stop 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 shutdown_connections 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f61c002ffb0 0x7f61c0032470 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 --2- 172.21.15.178:0/1611234385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61d4060280 0x7f61d41066e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 >> 172.21.15.178:0/1611234385 conn(0x7f61d40f0b70 msgr2=0x7f61d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 shutdown_connections 2024-01-21T11:18:43.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:43.389+0000 7f61dc743700 1 -- 172.21.15.178:0/1611234385 wait complete. 2024-01-21T11:18:43.390 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:44.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:43 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:43.388341+0000 mon.smithi089 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.178:0/1611234385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:44.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:44.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:45.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:43.689884+0000 mgr.smithi089.ufybqw (mgr.14162) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 -- 172.21.15.178:0/3404920647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 msgr2=0x7fe7c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 --2- 172.21.15.178:0/3404920647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c00f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe7b8009960 tx=0x7fe7b8004d10).stop 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 -- 172.21.15.178:0/3404920647 shutdown_connections 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 --2- 172.21.15.178:0/3404920647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c00f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 -- 172.21.15.178:0/3404920647 >> 172.21.15.178:0/3404920647 conn(0x7fe7c00f0b70 msgr2=0x7fe7c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 -- 172.21.15.178:0/3404920647 shutdown_connections 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.994+0000 7fe7c801d700 1 -- 172.21.15.178:0/3404920647 wait complete. 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c801d700 1 Processor -- start 2024-01-21T11:18:45.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c801d700 1 -- start start 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c801d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c801d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c0060220 con 0x7fe7c00f5410 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c5db9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c5db9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40782/0 (socket says 172.21.15.178:40782) 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c5db9700 1 -- 172.21.15.178:0/820711257 learned_addr learned my addr 172.21.15.178:0/820711257 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.995+0000 7fe7c5db9700 1 -- 172.21.15.178:0/820711257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b8009640 con 0x7fe7c00f5410 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.996+0000 7fe7c5db9700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c0109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe7b8004b70 tx=0x7fe7b8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.996+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8018070 con 0x7fe7c00f5410 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.996+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c0060420 con 0x7fe7c00f5410 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.996+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7b800b870 con 0x7fe7c00f5410 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.996+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b801c560 con 0x7fe7c00f5410 2024-01-21T11:18:45.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.996+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c0109b00 con 0x7fe7c00f5410 2024-01-21T11:18:45.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.997+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7b801c6c0 con 0x7fe7c00f5410 2024-01-21T11:18:45.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.997+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c0059100 con 0x7fe7c00f5410 2024-01-21T11:18:45.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.997+0000 7fe7b67fc700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe7ac02ff60 0x7fe7ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:45.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.997+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7b8041590 con 0x7fe7c00f5410 2024-01-21T11:18:45.999 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.997+0000 7fe7c55b8700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe7ac02ff60 0x7fe7ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:45.999 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:45.997+0000 7fe7c55b8700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe7ac02ff60 0x7fe7ac032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe7b00096a0 tx=0x7fe7b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:46.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.002+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7b8012560 con 0x7fe7c00f5410 2024-01-21T11:18:46.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:45.690192+0000 mgr.smithi089.ufybqw (mgr.14162) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:46.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.208+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7c0045ad0 con 0x7fe7c00f5410 2024-01-21T11:18:46.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.209+0000 7fe7b67fc700 1 -- 172.21.15.178:0/820711257 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7b8012380 con 0x7fe7c00f5410 2024-01-21T11:18:46.209 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:46.209 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:46.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.210+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe7ac02ff60 msgr2=0x7fe7ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:46.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.210+0000 7fe7c801d700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe7ac02ff60 0x7fe7ac032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe7b00096a0 tx=0x7fe7b0006b40).stop 2024-01-21T11:18:46.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.210+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 msgr2=0x7fe7c0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:46.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.210+0000 7fe7c801d700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c0109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe7b8004b70 tx=0x7fe7b8004d10).stop 2024-01-21T11:18:46.210 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.211+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 shutdown_connections 2024-01-21T11:18:46.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.211+0000 7fe7c801d700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe7ac02ff60 0x7fe7ac032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:46.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.211+0000 7fe7c801d700 1 --2- 172.21.15.178:0/820711257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c00f5410 0x7fe7c0109450 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:46.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.211+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 >> 172.21.15.178:0/820711257 conn(0x7fe7c00f0b70 msgr2=0x7fe7c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:46.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.211+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 shutdown_connections 2024-01-21T11:18:46.211 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:46.211+0000 7fe7c801d700 1 -- 172.21.15.178:0/820711257 wait complete. 2024-01-21T11:18:46.211 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:47.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:46.209887+0000 mon.smithi089 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.178:0/820711257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:47.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:47.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:48.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:47.690508+0000 mgr.smithi089.ufybqw (mgr.14162) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:48.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 -- 172.21.15.178:0/2952659581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 msgr2=0x7f42f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:48.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 --2- 172.21.15.178:0/2952659581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f40f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f42e4009fb0 tx=0x7f42e4009b30).stop 2024-01-21T11:18:48.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 -- 172.21.15.178:0/2952659581 shutdown_connections 2024-01-21T11:18:48.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 --2- 172.21.15.178:0/2952659581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f40f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:48.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 -- 172.21.15.178:0/2952659581 >> 172.21.15.178:0/2952659581 conn(0x7f42f40f0b70 msgr2=0x7f42f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:48.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 -- 172.21.15.178:0/2952659581 shutdown_connections 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.758+0000 7f42fb51a700 1 -- 172.21.15.178:0/2952659581 wait complete. 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42fb51a700 1 Processor -- start 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42fb51a700 1 -- start start 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42fb51a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42fb51a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4105b20 con 0x7f42f40f5410 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42f92b6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42f92b6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40790/0 (socket says 172.21.15.178:40790) 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42f92b6700 1 -- 172.21.15.178:0/1598843572 learned_addr learned my addr 172.21.15.178:0/1598843572 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.759+0000 7f42f92b6700 1 -- 172.21.15.178:0/1598843572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4005040 con 0x7f42f40f5410 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.760+0000 7f42f92b6700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f4105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f42e4014000 tx=0x7f42e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:48.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.760+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e40095f0 con 0x7f42f40f5410 2024-01-21T11:18:48.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.760+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f4105d20 con 0x7f42f40f5410 2024-01-21T11:18:48.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.760+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42e4007e70 con 0x7f42f40f5410 2024-01-21T11:18:48.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.760+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e4014ea0 con 0x7f42f40f5410 2024-01-21T11:18:48.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.760+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f41061b0 con 0x7f42f40f5410 2024-01-21T11:18:48.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.761+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42e4025070 con 0x7f42f40f5410 2024-01-21T11:18:48.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.761+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f4045ad0 con 0x7f42f40f5410 2024-01-21T11:18:48.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.761+0000 7f42e9ffb700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42e002ff60 0x7f42e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:48.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.761+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42e4040970 con 0x7f42f40f5410 2024-01-21T11:18:48.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.761+0000 7f42f8ab5700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42e002ff60 0x7f42e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:48.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.761+0000 7f42f8ab5700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42e002ff60 0x7f42e0032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f42f00096a0 tx=0x7f42f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:48.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.765+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42e40079c0 con 0x7f42f40f5410 2024-01-21T11:18:48.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.970+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42f40fae60 con 0x7f42f40f5410 2024-01-21T11:18:48.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.971+0000 7f42e9ffb700 1 -- 172.21.15.178:0/1598843572 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42e400f300 con 0x7f42f40f5410 2024-01-21T11:18:48.972 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:48.972 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:48.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42e002ff60 msgr2=0x7f42e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:48.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42e002ff60 0x7f42e0032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f42f00096a0 tx=0x7f42f0006b40).stop 2024-01-21T11:18:48.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 msgr2=0x7f42f4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f4105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f42e4014000 tx=0x7f42e400b040).stop 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 shutdown_connections 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42e002ff60 0x7f42e0032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 --2- 172.21.15.178:0/1598843572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42f40f5410 0x7f42f4105630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 >> 172.21.15.178:0/1598843572 conn(0x7f42f40f0b70 msgr2=0x7f42f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 shutdown_connections 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:48.972+0000 7f42fb51a700 1 -- 172.21.15.178:0/1598843572 wait complete. 2024-01-21T11:18:48.974 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:49.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:49 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:48.972635+0000 mon.smithi089 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.178:0/1598843572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:50.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:50.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:51.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:49.690711+0000 mgr.smithi089.ufybqw (mgr.14162) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:51.500 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 -- 172.21.15.178:0/637515706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 msgr2=0x7efdd40fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:51.500 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 --2- 172.21.15.178:0/637515706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd40fb550 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7efdc8009fb0 tx=0x7efdc8009b30).stop 2024-01-21T11:18:51.500 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 -- 172.21.15.178:0/637515706 shutdown_connections 2024-01-21T11:18:51.500 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 --2- 172.21.15.178:0/637515706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd40fb550 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:51.500 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 -- 172.21.15.178:0/637515706 >> 172.21.15.178:0/637515706 conn(0x7efdd40f0b70 msgr2=0x7efdd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 -- 172.21.15.178:0/637515706 shutdown_connections 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 -- 172.21.15.178:0/637515706 wait complete. 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 Processor -- start 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.498+0000 7efdd92ec700 1 -- start start 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd92ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd41088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd92ec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4108db0 con 0x7efdd4060220 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd41088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd41088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:40814/0 (socket says 172.21.15.178:40814) 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd2ffd700 1 -- 172.21.15.178:0/1462261906 learned_addr learned my addr 172.21.15.178:0/1462261906 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd2ffd700 1 -- 172.21.15.178:0/1462261906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8005040 con 0x7efdd4060220 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdd2ffd700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd41088c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7efdc8013510 tx=0x7efdc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.499+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc8009320 con 0x7efdd4060220 2024-01-21T11:18:51.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.500+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd4108fb0 con 0x7efdd4060220 2024-01-21T11:18:51.502 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.500+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdc8013f00 con 0x7efdd4060220 2024-01-21T11:18:51.502 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.500+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc8014dc0 con 0x7efdd4060220 2024-01-21T11:18:51.502 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.500+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd4109440 con 0x7efdd4060220 2024-01-21T11:18:51.502 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.500+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd4045ad0 con 0x7efdd4060220 2024-01-21T11:18:51.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.500+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdc80095f0 con 0x7efdd4060220 2024-01-21T11:18:51.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.501+0000 7efdbb7fe700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efdbc02ff60 0x7efdbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:51.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.501+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdc8041020 con 0x7efdd4060220 2024-01-21T11:18:51.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.501+0000 7efdd27fc700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efdbc02ff60 0x7efdbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:51.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.501+0000 7efdd27fc700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efdbc02ff60 0x7efdbc032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7efdc40096a0 tx=0x7efdc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:51.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.505+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdc8014420 con 0x7efdd4060220 2024-01-21T11:18:51.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.711+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdd410a290 con 0x7efdd4060220 2024-01-21T11:18:51.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.712+0000 7efdbb7fe700 1 -- 172.21.15.178:0/1462261906 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdc801c920 con 0x7efdd4060220 2024-01-21T11:18:51.713 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:51.713 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:51.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efdbc02ff60 msgr2=0x7efdbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:51.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efdbc02ff60 0x7efdbc032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7efdc40096a0 tx=0x7efdc4006b40).stop 2024-01-21T11:18:51.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 msgr2=0x7efdd41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:51.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd41088c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7efdc8013510 tx=0x7efdc800b040).stop 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 shutdown_connections 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7efdbc02ff60 0x7efdbc032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 --2- 172.21.15.178:0/1462261906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4060220 0x7efdd41088c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 >> 172.21.15.178:0/1462261906 conn(0x7efdd40f0b70 msgr2=0x7efdd40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 shutdown_connections 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:51.713+0000 7efdd92ec700 1 -- 172.21.15.178:0/1462261906 wait complete. 2024-01-21T11:18:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:53.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:53.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:53.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:51.691016+0000 mgr.smithi089.ufybqw (mgr.14162) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:53.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:51.713552+0000 mon.smithi089 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.178:0/1462261906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 -- 172.21.15.178:0/600330717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 msgr2=0x7f971c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 --2- 172.21.15.178:0/600330717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c0f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f9710009fb0 tx=0x7f9710009b30).stop 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 -- 172.21.15.178:0/600330717 shutdown_connections 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 --2- 172.21.15.178:0/600330717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c0f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 -- 172.21.15.178:0/600330717 >> 172.21.15.178:0/600330717 conn(0x7f971c0f0b70 msgr2=0x7f971c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 -- 172.21.15.178:0/600330717 shutdown_connections 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.267+0000 7f9721105700 1 -- 172.21.15.178:0/600330717 wait complete. 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f9721105700 1 Processor -- start 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f9721105700 1 -- start start 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f9721105700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:54.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f9721105700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c106c10 con 0x7f971c0f5410 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f971ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f971ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44250/0 (socket says 172.21.15.178:44250) 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f971ad9d700 1 -- 172.21.15.178:0/3383455730 learned_addr learned my addr 172.21.15.178:0/3383455730 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.268+0000 7f971ad9d700 1 -- 172.21.15.178:0/3383455730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9710005040 con 0x7f971c0f5410 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.269+0000 7f971ad9d700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c106720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f9710013510 tx=0x7f971000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.269+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9710009320 con 0x7f971c0f5410 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.269+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c106e10 con 0x7f971c0f5410 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.269+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9710013f00 con 0x7f971c0f5410 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.269+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9710014d90 con 0x7f971c0f5410 2024-01-21T11:18:54.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.269+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c1072a0 con 0x7f971c0f5410 2024-01-21T11:18:54.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.270+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c045ad0 con 0x7f971c0f5410 2024-01-21T11:18:54.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.270+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9710014420 con 0x7f971c0f5410 2024-01-21T11:18:54.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.270+0000 7f97037fe700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f970402ff60 0x7f9704032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:54.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.270+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9710041080 con 0x7f971c0f5410 2024-01-21T11:18:54.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.270+0000 7f971a59c700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f970402ff60 0x7f9704032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:54.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.270+0000 7f971a59c700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f970402ff60 0x7f9704032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f970c0096a0 tx=0x7f970c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:54.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.274+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97100095f0 con 0x7f971c0f5410 2024-01-21T11:18:54.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.480+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f971c108150 con 0x7f971c0f5410 2024-01-21T11:18:54.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.481+0000 7f97037fe700 1 -- 172.21.15.178:0/3383455730 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f971001db70 con 0x7f971c0f5410 2024-01-21T11:18:54.482 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:54.482 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f970402ff60 msgr2=0x7f9704032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f970402ff60 0x7f9704032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f970c0096a0 tx=0x7f970c006b40).stop 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 msgr2=0x7f971c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c106720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f9710013510 tx=0x7f971000b040).stop 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 shutdown_connections 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f970402ff60 0x7f9704032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 --2- 172.21.15.178:0/3383455730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f971c0f5410 0x7f971c106720 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 >> 172.21.15.178:0/3383455730 conn(0x7f971c0f0b70 msgr2=0x7f971c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 shutdown_connections 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:54.482+0000 7f9721105700 1 -- 172.21.15.178:0/3383455730 wait complete. 2024-01-21T11:18:54.484 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:55.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:53.691367+0000 mgr.smithi089.ufybqw (mgr.14162) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:55.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:54.482423+0000 mon.smithi089 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.178:0/3383455730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:55.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:55.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:57.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 -- 172.21.15.178:0/1539040083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0f51a0 msgr2=0x7fbabc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:57.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 --2- 172.21.15.178:0/1539040083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0f51a0 0x7fbabc0f55c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fbaac009fb0 tx=0x7fbaac009b30).stop 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 -- 172.21.15.178:0/1539040083 shutdown_connections 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 --2- 172.21.15.178:0/1539040083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0f51a0 0x7fbabc0f55c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 -- 172.21.15.178:0/1539040083 >> 172.21.15.178:0/1539040083 conn(0x7fbabc0f09a0 msgr2=0x7fbabc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 -- 172.21.15.178:0/1539040083 shutdown_connections 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 -- 172.21.15.178:0/1539040083 wait complete. 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.030+0000 7fbac159f700 1 Processor -- start 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbac159f700 1 -- start start 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbac159f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 0x7fbabc109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbac159f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc109850 con 0x7fbabc0601e0 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbabbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 0x7fbabc109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbabbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 0x7fbabc109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44272/0 (socket says 172.21.15.178:44272) 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbabbfff700 1 -- 172.21.15.178:0/2738202791 learned_addr learned my addr 172.21.15.178:0/2738202791 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbabbfff700 1 -- 172.21.15.178:0/2738202791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac005040 con 0x7fbabc0601e0 2024-01-21T11:18:57.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.031+0000 7fbabbfff700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 0x7fbabc109360 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fbaac009f80 tx=0x7fbaac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:57.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaac0136c0 con 0x7fbabc0601e0 2024-01-21T11:18:57.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc109990 con 0x7fbabc0601e0 2024-01-21T11:18:57.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbaac013e70 con 0x7fbabc0601e0 2024-01-21T11:18:57.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaac014420 con 0x7fbabc0601e0 2024-01-21T11:18:57.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc109dc0 con 0x7fbabc0601e0 2024-01-21T11:18:57.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbaac025060 con 0x7fbabc0601e0 2024-01-21T11:18:57.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.032+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc045ad0 con 0x7fbabc0601e0 2024-01-21T11:18:57.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.033+0000 7fbab8ff9700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbaa802ff60 0x7fbaa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:57.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.033+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbaac040f30 con 0x7fbabc0601e0 2024-01-21T11:18:57.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.033+0000 7fbabb7fe700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbaa802ff60 0x7fbaa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:57.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.033+0000 7fbabb7fe700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbaa802ff60 0x7fbaa8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fbaa40096a0 tx=0x7fbaa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:57.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.037+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbaac01d3a0 con 0x7fbabc0601e0 2024-01-21T11:18:57.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:55.691705+0000 mgr.smithi089.ufybqw (mgr.14162) 172 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:57.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.243+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbabc0fabe0 con 0x7fbabc0601e0 2024-01-21T11:18:57.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.244+0000 7fbab8ff9700 1 -- 172.21.15.178:0/2738202791 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbaac004490 con 0x7fbabc0601e0 2024-01-21T11:18:57.245 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:18:57.245 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:18:57.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbaa802ff60 msgr2=0x7fbaa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:57.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbaa802ff60 0x7fbaa8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fbaa40096a0 tx=0x7fbaa4006b40).stop 2024-01-21T11:18:57.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 msgr2=0x7fbabc109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:57.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 0x7fbabc109360 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fbaac009f80 tx=0x7fbaac009b30).stop 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 shutdown_connections 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbaa802ff60 0x7fbaa8032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 --2- 172.21.15.178:0/2738202791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbabc0601e0 0x7fbabc109360 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.245+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 >> 172.21.15.178:0/2738202791 conn(0x7fbabc0f09a0 msgr2=0x7fbabc0f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.246+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 shutdown_connections 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:57.246+0000 7fbac159f700 1 -- 172.21.15.178:0/2738202791 wait complete. 2024-01-21T11:18:57.247 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:18:58.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:57 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:18:57.245687+0000 mon.smithi089 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.178:0/2738202791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:18:58.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:18:58.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:18:59.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:18:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:57.692012+0000 mgr.smithi089.ufybqw (mgr.14162) 173 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:18:59.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 -- 172.21.15.178:0/817245027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 msgr2=0x7fc6300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 --2- 172.21.15.178:0/817245027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6300f66c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc624009fb0 tx=0x7fc624009b30).stop 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 -- 172.21.15.178:0/817245027 shutdown_connections 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 --2- 172.21.15.178:0/817245027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6300f66c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 -- 172.21.15.178:0/817245027 >> 172.21.15.178:0/817245027 conn(0x7fc6300f1aa0 msgr2=0x7fc6300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 -- 172.21.15.178:0/817245027 shutdown_connections 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 -- 172.21.15.178:0/817245027 wait complete. 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 Processor -- start 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.848+0000 7fc635f06700 1 -- start start 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.849+0000 7fc635f06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6301064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:59.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.849+0000 7fc635f06700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6301069a0 con 0x7fc6300f62a0 2024-01-21T11:18:59.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.849+0000 7fc634f04700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6301064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:59.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.849+0000 7fc634f04700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6301064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:44284/0 (socket says 172.21.15.178:44284) 2024-01-21T11:18:59.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.849+0000 7fc634f04700 1 -- 172.21.15.178:0/1857320652 learned_addr learned my addr 172.21.15.178:0/1857320652 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.849+0000 7fc634f04700 1 -- 172.21.15.178:0/1857320652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc624005040 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.850+0000 7fc634f04700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6301064b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fc624013510 tx=0x7fc62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.850+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc624009320 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.850+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc630106ba0 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.850+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc624013f00 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.850+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc624014dc0 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.850+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc630107030 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.851+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc62401d3d0 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.851+0000 7fc62d7fa700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc62002ff60 0x7fc620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.851+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6240410b0 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.851+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc630045ad0 con 0x7fc6300f62a0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.851+0000 7fc62ffff700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc62002ff60 0x7fc620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:18:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.851+0000 7fc62ffff700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc62002ff60 0x7fc620032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc61c0096a0 tx=0x7fc61c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:18:59.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:18:59.856+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc624012900 con 0x7fc6300f62a0 2024-01-21T11:19:00.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.061+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc630107d80 con 0x7fc6300f62a0 2024-01-21T11:19:00.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.063+0000 7fc62d7fa700 1 -- 172.21.15.178:0/1857320652 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc62402b020 con 0x7fc6300f62a0 2024-01-21T11:19:00.063 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:00.063 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:00.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.063+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc62002ff60 msgr2=0x7fc620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:00.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.063+0000 7fc635f06700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc62002ff60 0x7fc620032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc61c0096a0 tx=0x7fc61c006b40).stop 2024-01-21T11:19:00.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.063+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 msgr2=0x7fc6301064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.063+0000 7fc635f06700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6301064b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fc624013510 tx=0x7fc62400b040).stop 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.064+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 shutdown_connections 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.064+0000 7fc635f06700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc62002ff60 0x7fc620032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.064+0000 7fc635f06700 1 --2- 172.21.15.178:0/1857320652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6300f62a0 0x7fc6301064b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.064+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 >> 172.21.15.178:0/1857320652 conn(0x7fc6300f1aa0 msgr2=0x7fc6300f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.064+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 shutdown_connections 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:00.064+0000 7fc635f06700 1 -- 172.21.15.178:0/1857320652 wait complete. 2024-01-21T11:19:00.065 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:01.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:18:59.692296+0000 mgr.smithi089.ufybqw (mgr.14162) 174 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:01.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:00.063700+0000 mon.smithi089 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.178:0/1857320652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:01.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:01.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:02.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 -- 172.21.15.178:0/729995449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 msgr2=0x7ff3200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 --2- 172.21.15.178:0/729995449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff3200f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff308009fb0 tx=0x7ff308009b30).stop 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 -- 172.21.15.178:0/729995449 shutdown_connections 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 --2- 172.21.15.178:0/729995449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff3200f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 -- 172.21.15.178:0/729995449 >> 172.21.15.178:0/729995449 conn(0x7ff3200f0b70 msgr2=0x7ff3200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 -- 172.21.15.178:0/729995449 shutdown_connections 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.583+0000 7ff3266dd700 1 -- 172.21.15.178:0/729995449 wait complete. 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.584+0000 7ff3266dd700 1 Processor -- start 2024-01-21T11:19:02.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.584+0000 7ff3266dd700 1 -- start start 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.584+0000 7ff3266dd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff320105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.584+0000 7ff3266dd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff320105b20 con 0x7ff3200f5410 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff320105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff320105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43032/0 (socket says 172.21.15.178:43032) 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31ffff700 1 -- 172.21.15.178:0/4051227312 learned_addr learned my addr 172.21.15.178:0/4051227312 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31ffff700 1 -- 172.21.15.178:0/4051227312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff308005040 con 0x7ff3200f5410 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31ffff700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff320105630 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff308013510 tx=0x7ff30800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff308009320 con 0x7ff3200f5410 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff308013f00 con 0x7ff3200f5410 2024-01-21T11:19:02.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff320105d20 con 0x7ff3200f5410 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff308014d90 con 0x7ff3200f5410 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.585+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3201061b0 con 0x7ff3200f5410 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.586+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3080095f0 con 0x7ff3200f5410 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.586+0000 7ff31cff9700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff30c02ff60 0x7ff30c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.586+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff320045ad0 con 0x7ff3200f5410 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.586+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff308041000 con 0x7ff3200f5410 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.586+0000 7ff31f7fe700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff30c02ff60 0x7ff30c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.587+0000 7ff31f7fe700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff30c02ff60 0x7ff30c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff3100096a0 tx=0x7ff310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:02.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.591+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff30806d2b0 con 0x7ff3200f5410 2024-01-21T11:19:02.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.798+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3200faf50 con 0x7ff3200f5410 2024-01-21T11:19:02.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.799+0000 7ff31cff9700 1 -- 172.21.15.178:0/4051227312 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff308014420 con 0x7ff3200f5410 2024-01-21T11:19:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:02.800 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff30c02ff60 msgr2=0x7ff30c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff30c02ff60 0x7ff30c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff3100096a0 tx=0x7ff310006b40).stop 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 msgr2=0x7ff320105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff320105630 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff308013510 tx=0x7ff30800b040).stop 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 shutdown_connections 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff30c02ff60 0x7ff30c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:02.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 --2- 172.21.15.178:0/4051227312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3200f5410 0x7ff320105630 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:02.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 >> 172.21.15.178:0/4051227312 conn(0x7ff3200f0b70 msgr2=0x7ff3200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:02.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 shutdown_connections 2024-01-21T11:19:02.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:02.800+0000 7ff3266dd700 1 -- 172.21.15.178:0/4051227312 wait complete. 2024-01-21T11:19:02.802 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:01.692642+0000 mgr.smithi089.ufybqw (mgr.14162) 175 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:04.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:04.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:04.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:03 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:02.800427+0000 mon.smithi089 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.178:0/4051227312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:05.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:03.692884+0000 mgr.smithi089.ufybqw (mgr.14162) 176 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 -- 172.21.15.178:0/1989141709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 msgr2=0x7f62400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 --2- 172.21.15.178:0/1989141709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62400f55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6230009fb0 tx=0x7f6230009b30).stop 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 -- 172.21.15.178:0/1989141709 shutdown_connections 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 --2- 172.21.15.178:0/1989141709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62400f55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 -- 172.21.15.178:0/1989141709 >> 172.21.15.178:0/1989141709 conn(0x7f62400f09a0 msgr2=0x7f62400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 -- 172.21.15.178:0/1989141709 shutdown_connections 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.321+0000 7f6246801700 1 -- 172.21.15.178:0/1989141709 wait complete. 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f6246801700 1 Processor -- start 2024-01-21T11:19:05.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f6246801700 1 -- start start 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f6246801700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62401064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f6246801700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401069d0 con 0x7f62400f51a0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f62457ff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62401064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f62457ff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62401064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43050/0 (socket says 172.21.15.178:43050) 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.322+0000 7f62457ff700 1 -- 172.21.15.178:0/1131615029 learned_addr learned my addr 172.21.15.178:0/1131615029 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f62457ff700 1 -- 172.21.15.178:0/1131615029 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6230005040 con 0x7f62400f51a0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f62457ff700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62401064e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6230009830 tx=0x7f623000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6230009320 con 0x7f62400f51a0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6240106bd0 con 0x7f62400f51a0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62300136c0 con 0x7f62400f51a0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f623001ee90 con 0x7f62400f51a0 2024-01-21T11:19:05.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.323+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6240107060 con 0x7f62400f51a0 2024-01-21T11:19:05.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.324+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f623001e420 con 0x7f62400f51a0 2024-01-21T11:19:05.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.324+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6240045ad0 con 0x7f62400f51a0 2024-01-21T11:19:05.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.324+0000 7f6235ffb700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f622c02ff10 0x7f622c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:05.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.324+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62300410b0 con 0x7f62400f51a0 2024-01-21T11:19:05.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.324+0000 7f6244ffe700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f622c02ff10 0x7f622c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:05.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.325+0000 7f6244ffe700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f622c02ff10 0x7f622c0323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f623c0096a0 tx=0x7f623c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:05.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.329+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62300095f0 con 0x7f62400f51a0 2024-01-21T11:19:05.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.534+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6240107eb0 con 0x7f62400f51a0 2024-01-21T11:19:05.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.535+0000 7f6235ffb700 1 -- 172.21.15.178:0/1131615029 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f623002c2d0 con 0x7f62400f51a0 2024-01-21T11:19:05.536 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:05.536 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:05.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f622c02ff10 msgr2=0x7f622c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:05.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f622c02ff10 0x7f622c0323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f623c0096a0 tx=0x7f623c006b40).stop 2024-01-21T11:19:05.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 msgr2=0x7f62401064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:05.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62401064e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6230009830 tx=0x7f623000b040).stop 2024-01-21T11:19:05.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 shutdown_connections 2024-01-21T11:19:05.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f622c02ff10 0x7f622c0323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:05.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 --2- 172.21.15.178:0/1131615029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62400f51a0 0x7f62401064e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:05.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 >> 172.21.15.178:0/1131615029 conn(0x7f62400f09a0 msgr2=0x7f624005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:05.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 shutdown_connections 2024-01-21T11:19:05.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:05.536+0000 7f6246801700 1 -- 172.21.15.178:0/1131615029 wait complete. 2024-01-21T11:19:05.538 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:06.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:05.536528+0000 mon.smithi089 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.178:0/1131615029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:06.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:06.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:07.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:05.693202+0000 mgr.smithi089.ufybqw (mgr.14162) 177 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:08.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 -- 172.21.15.178:0/1917427939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 msgr2=0x7f929c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:08.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 --2- 172.21.15.178:0/1917427939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c0f4be0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f928c009fb0 tx=0x7f928c009b30).stop 2024-01-21T11:19:08.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 -- 172.21.15.178:0/1917427939 shutdown_connections 2024-01-21T11:19:08.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 --2- 172.21.15.178:0/1917427939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c0f4be0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:08.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 -- 172.21.15.178:0/1917427939 >> 172.21.15.178:0/1917427939 conn(0x7f929c0f02e0 msgr2=0x7f929c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:08.179 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 -- 172.21.15.178:0/1917427939 shutdown_connections 2024-01-21T11:19:08.180 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.178+0000 7f92a3b16700 1 -- 172.21.15.178:0/1917427939 wait complete. 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a3b16700 1 Processor -- start 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a3b16700 1 -- start start 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a3b16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a3b16700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c106bf0 con 0x7f929c0f47c0 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a18b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a18b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43072/0 (socket says 172.21.15.178:43072) 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a18b2700 1 -- 172.21.15.178:0/2314558514 learned_addr learned my addr 172.21.15.178:0/2314558514 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a18b2700 1 -- 172.21.15.178:0/2314558514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c005040 con 0x7f929c0f47c0 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.179+0000 7f92a18b2700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c106700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f928c013510 tx=0x7f928c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:08.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.180+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f928c009320 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.180+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c106df0 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.180+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f928c013f00 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.180+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f928c014d90 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.180+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c107280 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.180+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f928c014420 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.181+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c045ad0 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.181+0000 7f92927fc700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f928802ff60 0x7f9288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.181+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f928c041110 con 0x7f929c0f47c0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.181+0000 7f92a10b1700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f928802ff60 0x7f9288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:08.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.181+0000 7f92a10b1700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f928802ff60 0x7f9288032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:08.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.185+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f928c0095f0 con 0x7f929c0f47c0 2024-01-21T11:19:08.392 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.391+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f929c108130 con 0x7f929c0f47c0 2024-01-21T11:19:08.392 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.392+0000 7f92927fc700 1 -- 172.21.15.178:0/2314558514 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f928c01db70 con 0x7f929c0f47c0 2024-01-21T11:19:08.392 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:08.393 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:08.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f928802ff60 msgr2=0x7f9288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:08.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f928802ff60 0x7f9288032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).stop 2024-01-21T11:19:08.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 msgr2=0x7f929c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:08.394 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c106700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f928c013510 tx=0x7f928c00b040).stop 2024-01-21T11:19:08.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 shutdown_connections 2024-01-21T11:19:08.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f928802ff60 0x7f9288032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:08.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 --2- 172.21.15.178:0/2314558514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f929c0f47c0 0x7f929c106700 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:08.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 >> 172.21.15.178:0/2314558514 conn(0x7f929c0f02e0 msgr2=0x7f929c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:08.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 shutdown_connections 2024-01-21T11:19:08.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:08.393+0000 7f92a3b16700 1 -- 172.21.15.178:0/2314558514 wait complete. 2024-01-21T11:19:08.396 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:09.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:07.693590+0000 mgr.smithi089.ufybqw (mgr.14162) 178 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:09.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:08.393402+0000 mon.smithi089 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.178:0/2314558514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:09.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:09.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:10.938 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- 172.21.15.178:0/2516872650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 msgr2=0x7f45cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 --2- 172.21.15.178:0/2516872650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc0f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f45c0009fb0 tx=0x7f45c0009b30).stop 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- 172.21.15.178:0/2516872650 shutdown_connections 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 --2- 172.21.15.178:0/2516872650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc0f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- 172.21.15.178:0/2516872650 >> 172.21.15.178:0/2516872650 conn(0x7f45cc0f0b70 msgr2=0x7f45cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- 172.21.15.178:0/2516872650 shutdown_connections 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- 172.21.15.178:0/2516872650 wait complete. 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 Processor -- start 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- start start 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.936+0000 7f45d11a4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc060240 con 0x7f45cc0f5410 2024-01-21T11:19:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45cad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45cad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:43086/0 (socket says 172.21.15.178:43086) 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45cad9d700 1 -- 172.21.15.178:0/2776776391 learned_addr learned my addr 172.21.15.178:0/2776776391 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45cad9d700 1 -- 172.21.15.178:0/2776776391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c0005040 con 0x7f45cc0f5410 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45cad9d700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc109430 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f45c0013510 tx=0x7f45c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45c0009320 con 0x7f45cc0f5410 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc060440 con 0x7f45cc0f5410 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45c0013f00 con 0x7f45cc0f5410 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45c0014d90 con 0x7f45cc0f5410 2024-01-21T11:19:10.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.937+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc109ae0 con 0x7f45cc0f5410 2024-01-21T11:19:10.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.938+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45c00263a0 con 0x7f45cc0f5410 2024-01-21T11:19:10.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.938+0000 7f45b37fe700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:10.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.938+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45c0041010 con 0x7f45cc0f5410 2024-01-21T11:19:10.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.938+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc045ad0 con 0x7f45cc0f5410 2024-01-21T11:19:10.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.938+0000 7f45ca59c700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:10.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.939+0000 7f45ca59c700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f45b402ff60 0x7f45b4032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f45bc0096a0 tx=0x7f45bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:10.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:10.944+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45c0012920 con 0x7f45cc0f5410 2024-01-21T11:19:11.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.149+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45cc10a230 con 0x7f45cc0f5410 2024-01-21T11:19:11.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.150+0000 7f45b37fe700 1 -- 172.21.15.178:0/2776776391 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45c002b020 con 0x7f45cc0f5410 2024-01-21T11:19:11.152 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:11.152 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.151+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f45b402ff60 msgr2=0x7f45b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.151+0000 7f45d11a4700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f45b402ff60 0x7f45b4032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f45bc0096a0 tx=0x7f45bc006b40).stop 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.151+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 msgr2=0x7f45cc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.151+0000 7f45d11a4700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc109430 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f45c0013510 tx=0x7f45c000b040).stop 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.152+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 shutdown_connections 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.152+0000 7f45d11a4700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.152+0000 7f45d11a4700 1 --2- 172.21.15.178:0/2776776391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45cc0f5410 0x7f45cc109430 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.152+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 >> 172.21.15.178:0/2776776391 conn(0x7f45cc0f0b70 msgr2=0x7f45cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.152+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 shutdown_connections 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:11.152+0000 7f45d11a4700 1 -- 172.21.15.178:0/2776776391 wait complete. 2024-01-21T11:19:11.153 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:11.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:09.693863+0000 mgr.smithi089.ufybqw (mgr.14162) 179 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:12.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:11 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:11.151635+0000 mon.smithi089 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.178:0/2776776391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:12.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:12.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:13.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:11.694172+0000 mgr.smithi089.ufybqw (mgr.14162) 180 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 -- 172.21.15.178:0/3997602838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 msgr2=0x7f77dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 --2- 172.21.15.178:0/3997602838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc0f55c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f77cc009fb0 tx=0x7f77cc009b30).stop 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 -- 172.21.15.178:0/3997602838 shutdown_connections 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 --2- 172.21.15.178:0/3997602838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc0f55c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 -- 172.21.15.178:0/3997602838 >> 172.21.15.178:0/3997602838 conn(0x7f77dc0f09a0 msgr2=0x7f77dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 -- 172.21.15.178:0/3997602838 shutdown_connections 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 -- 172.21.15.178:0/3997602838 wait complete. 2024-01-21T11:19:13.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 Processor -- start 2024-01-21T11:19:13.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.714+0000 7f77e134a700 1 -- start start 2024-01-21T11:19:13.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77e134a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:13.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77e134a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc1069d0 con 0x7f77dc0f51a0 2024-01-21T11:19:13.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:13.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42650/0 (socket says 172.21.15.178:42650) 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77dbfff700 1 -- 172.21.15.178:0/2845596360 learned_addr learned my addr 172.21.15.178:0/2845596360 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77dbfff700 1 -- 172.21.15.178:0/2845596360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc005040 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77dbfff700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc1064e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f77cc013510 tx=0x7f77cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77cc009320 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77dc106bd0 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77cc013f00 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77cc014dc0 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.715+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77dc107060 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.716+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77cc0095f0 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.716+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77dc045ad0 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.716+0000 7f77d8ff9700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77c402ff60 0x7f77c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.716+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77cc041100 con 0x7f77dc0f51a0 2024-01-21T11:19:13.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.717+0000 7f77db7fe700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77c402ff60 0x7f77c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:13.718 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.717+0000 7f77db7fe700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77c402ff60 0x7f77c4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f77d00096a0 tx=0x7f77d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:13.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.721+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77cc014420 con 0x7f77dc0f51a0 2024-01-21T11:19:13.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.928+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77dc107f10 con 0x7f77dc0f51a0 2024-01-21T11:19:13.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.929+0000 7f77d8ff9700 1 -- 172.21.15.178:0/2845596360 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77cc010230 con 0x7f77dc0f51a0 2024-01-21T11:19:13.929 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:13.929 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:13.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77c402ff60 msgr2=0x7f77c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77c402ff60 0x7f77c4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f77d00096a0 tx=0x7f77d0006b40).stop 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 msgr2=0x7f77dc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc1064e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f77cc013510 tx=0x7f77cc00b040).stop 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 shutdown_connections 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f77c402ff60 0x7f77c4032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 --2- 172.21.15.178:0/2845596360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc0f51a0 0x7f77dc1064e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.930+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 >> 172.21.15.178:0/2845596360 conn(0x7f77dc0f09a0 msgr2=0x7f77dc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.931+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 shutdown_connections 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:13.931+0000 7f77e134a700 1 -- 172.21.15.178:0/2845596360 wait complete. 2024-01-21T11:19:13.931 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:13.694421+0000 mgr.smithi089.ufybqw (mgr.14162) 181 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:13.776606+0000 mon.smithi089 (mon.0) 244 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:19:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:13.785222+0000 mon.smithi089 (mon.0) 245 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:19:15.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:13.929311+0000 mon.smithi089 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.178:0/2845596360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:15.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:15.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 -- 172.21.15.178:0/1497125962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 msgr2=0x7fdfd40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 --2- 172.21.15.178:0/1497125962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f90f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fdfc8009fb0 tx=0x7fdfc8009b30).stop 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 -- 172.21.15.178:0/1497125962 shutdown_connections 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 --2- 172.21.15.178:0/1497125962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f90f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 -- 172.21.15.178:0/1497125962 >> 172.21.15.178:0/1497125962 conn(0x7fdfd40f0b70 msgr2=0x7fdfd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 -- 172.21.15.178:0/1497125962 shutdown_connections 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.569+0000 7fdfd894e700 1 -- 172.21.15.178:0/1497125962 wait complete. 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd894e700 1 Processor -- start 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd894e700 1 -- start start 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd894e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:16.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd894e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40f4ff0 con 0x7fdfd4060280 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42658/0 (socket says 172.21.15.178:42658) 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.570+0000 7fdfd259c700 1 -- 172.21.15.178:0/1179997685 learned_addr learned my addr 172.21.15.178:0/1179997685 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfd259c700 1 -- 172.21.15.178:0/1179997685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc8005040 con 0x7fdfd4060280 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfd259c700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f6b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fdfc8013510 tx=0x7fdfc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfc8009320 con 0x7fdfd4060280 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd40f5190 con 0x7fdfd4060280 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfc8013f00 con 0x7fdfd4060280 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfc8014dc0 con 0x7fdfd4060280 2024-01-21T11:19:16.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.571+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd40f5620 con 0x7fdfd4060280 2024-01-21T11:19:16.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.572+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfc8014420 con 0x7fdfd4060280 2024-01-21T11:19:16.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.572+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd4059100 con 0x7fdfd4060280 2024-01-21T11:19:16.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.572+0000 7fdfc2ffd700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfbc02ff60 0x7fdfbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:16.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.572+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfc8041110 con 0x7fdfd4060280 2024-01-21T11:19:16.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.572+0000 7fdfd1d9b700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfbc02ff60 0x7fdfbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:16.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.572+0000 7fdfd1d9b700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfbc02ff60 0x7fdfbc032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fdfc40096a0 tx=0x7fdfc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:16.576 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.577+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfc80095f0 con 0x7fdfd4060280 2024-01-21T11:19:16.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.782+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfd40fbf50 con 0x7fdfd4060280 2024-01-21T11:19:16.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.783+0000 7fdfc2ffd700 1 -- 172.21.15.178:0/1179997685 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfc801cb10 con 0x7fdfd4060280 2024-01-21T11:19:16.782 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:16.783 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:16.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfbc02ff60 msgr2=0x7fdfbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:16.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfbc02ff60 0x7fdfbc032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fdfc40096a0 tx=0x7fdfc4006b40).stop 2024-01-21T11:19:16.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 msgr2=0x7fdfd40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:16.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f6b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fdfc8013510 tx=0x7fdfc800b040).stop 2024-01-21T11:19:16.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 shutdown_connections 2024-01-21T11:19:16.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fdfbc02ff60 0x7fdfbc032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:16.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 --2- 172.21.15.178:0/1179997685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd4060280 0x7fdfd40f6b70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:16.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 >> 172.21.15.178:0/1179997685 conn(0x7fdfd40f0b70 msgr2=0x7fdfd40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:16.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 shutdown_connections 2024-01-21T11:19:16.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:16.784+0000 7fdfd894e700 1 -- 172.21.15.178:0/1179997685 wait complete. 2024-01-21T11:19:16.785 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:17.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:15.694766+0000 mgr.smithi089.ufybqw (mgr.14162) 182 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:18.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:18.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:18.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:16.783448+0000 mon.smithi089 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.178:0/1179997685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:19.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:17.695134+0000 mgr.smithi089.ufybqw (mgr.14162) 183 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:19.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.329+0000 7f68ba37f700 1 -- 172.21.15.178:0/113929230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 msgr2=0x7f68b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:19.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.329+0000 7f68ba37f700 1 --2- 172.21.15.178:0/113929230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b40f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f68a4009fb0 tx=0x7f68a4009b30).stop 2024-01-21T11:19:19.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.329+0000 7f68ba37f700 1 -- 172.21.15.178:0/113929230 shutdown_connections 2024-01-21T11:19:19.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.329+0000 7f68ba37f700 1 --2- 172.21.15.178:0/113929230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b40f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:19.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.329+0000 7f68ba37f700 1 -- 172.21.15.178:0/113929230 >> 172.21.15.178:0/113929230 conn(0x7f68b40f0b70 msgr2=0x7f68b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:19.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.329+0000 7f68ba37f700 1 -- 172.21.15.178:0/113929230 shutdown_connections 2024-01-21T11:19:19.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.330+0000 7f68ba37f700 1 -- 172.21.15.178:0/113929230 wait complete. 2024-01-21T11:19:19.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.330+0000 7f68ba37f700 1 Processor -- start 2024-01-21T11:19:19.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.330+0000 7f68ba37f700 1 -- start start 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68ba37f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68ba37f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4105b20 con 0x7f68b40f5410 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42668/0 (socket says 172.21.15.178:42668) 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b3fff700 1 -- 172.21.15.178:0/3690907762 learned_addr learned my addr 172.21.15.178:0/3690907762 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b3fff700 1 -- 172.21.15.178:0/3690907762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a4005040 con 0x7f68b40f5410 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b3fff700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b4105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f68a4013510 tx=0x7f68a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a4009320 con 0x7f68b40f5410 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4105d20 con 0x7f68b40f5410 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68a4013f00 con 0x7f68b40f5410 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a4014d90 con 0x7f68b40f5410 2024-01-21T11:19:19.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.331+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b41061b0 con 0x7f68b40f5410 2024-01-21T11:19:19.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.332+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68a40263a0 con 0x7f68b40f5410 2024-01-21T11:19:19.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.332+0000 7f68b0ff9700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f689c02ff60 0x7f689c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:19.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.332+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68a4041010 con 0x7f68b40f5410 2024-01-21T11:19:19.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.332+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b4059100 con 0x7f68b40f5410 2024-01-21T11:19:19.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.333+0000 7f68b37fe700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f689c02ff60 0x7f689c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:19.334 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.333+0000 7f68b37fe700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f689c02ff60 0x7f689c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f68a80096a0 tx=0x7f68a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:19.339 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.337+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68a4012920 con 0x7f68b40f5410 2024-01-21T11:19:19.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.542+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68b40fae60 con 0x7f68b40f5410 2024-01-21T11:19:19.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.542+0000 7f68b0ff9700 1 -- 172.21.15.178:0/3690907762 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f68a402b020 con 0x7f68b40f5410 2024-01-21T11:19:19.543 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:19.543 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f689c02ff60 msgr2=0x7f689c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f689c02ff60 0x7f689c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f68a80096a0 tx=0x7f68a8006b40).stop 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 msgr2=0x7f68b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b4105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f68a4013510 tx=0x7f68a400b040).stop 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 shutdown_connections 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f689c02ff60 0x7f689c032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 --2- 172.21.15.178:0/3690907762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68b40f5410 0x7f68b4105630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 >> 172.21.15.178:0/3690907762 conn(0x7f68b40f0b70 msgr2=0x7f68b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:19.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 shutdown_connections 2024-01-21T11:19:19.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:19.544+0000 7f68ba37f700 1 -- 172.21.15.178:0/3690907762 wait complete. 2024-01-21T11:19:19.545 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:20.178 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:19 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:19.543265+0000 mon.smithi089 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.178:0/3690907762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:20.792 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:19.695400+0000 mgr.smithi089.ufybqw (mgr.14162) 184 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:20.793 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:20.180161+0000 mon.smithi089 (mon.0) 249 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T11:19:20.793 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:20.183289+0000 mon.smithi089 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T11:19:20.793 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:20.184046+0000 mon.smithi089 (mon.0) 251 : audit [DBG] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T11:19:20.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:20.914 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:21.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:19:20.178861+0000 mgr.smithi089.ufybqw (mgr.14162) 185 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi089 ... 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:14:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:15:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:15:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:16:57Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:18:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:18:42Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Traceback (most recent call last): 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: main() 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: r = ctx.func(ctx) 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: return func(ctx) 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T11:19:21.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: raise RuntimeError('uid/gid not found') 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: RuntimeError: uid/gid not found 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Traceback (most recent call last): 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: yield (conn, connr) 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: code, '\n'.join(err))) 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi089 ... 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:14:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.937 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:15:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:15:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi089 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:16:57Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:18:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr time="2024-01-21T11:18:42Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Traceback (most recent call last): 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: main() 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:19:21.938 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: r = ctx.func(ctx) 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: return func(ctx) 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: raise RuntimeError('uid/gid not found') 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: RuntimeError: uid/gid not found 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:19:20.184598+0000 mgr.smithi089.ufybqw (mgr.14162) 186 : cephadm [INF] Deploying daemon crash.smithi089 on smithi089 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:21.535784+0000 mon.smithi089 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:21.616147+0000 mon.smithi089 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:21.618334+0000 mon.smithi089 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:19:21.939 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:21.618729+0000 mon.smithi089 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T11:19:21.940 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:21.621336+0000 mon.smithi089 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:19:22.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 -- 172.21.15.178:0/3590099870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 msgr2=0x7f14840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 --2- 172.21.15.178:0/3590099870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14840f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1478009fb0 tx=0x7f1478009b30).stop 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 -- 172.21.15.178:0/3590099870 shutdown_connections 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 --2- 172.21.15.178:0/3590099870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14840f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 -- 172.21.15.178:0/3590099870 >> 172.21.15.178:0/3590099870 conn(0x7f14840f0b70 msgr2=0x7f14840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 -- 172.21.15.178:0/3590099870 shutdown_connections 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 -- 172.21.15.178:0/3590099870 wait complete. 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 Processor -- start 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.017+0000 7f148b242700 1 -- start start 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f148b242700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14841093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f148b242700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1484060240 con 0x7f14840f5410 2024-01-21T11:19:22.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f1488fde700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14841093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f1488fde700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14841093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:42690/0 (socket says 172.21.15.178:42690) 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f1488fde700 1 -- 172.21.15.178:0/2016174326 learned_addr learned my addr 172.21.15.178:0/2016174326 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f1488fde700 1 -- 172.21.15.178:0/2016174326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478005040 con 0x7f14840f5410 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f1488fde700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14841093d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1478013510 tx=0x7f147800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1478009320 con 0x7f14840f5410 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1484060440 con 0x7f14840f5410 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1478013f00 con 0x7f14840f5410 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1478014d90 con 0x7f14840f5410 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.018+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1484109a80 con 0x7f14840f5410 2024-01-21T11:19:22.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.019+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14780263a0 con 0x7f14840f5410 2024-01-21T11:19:22.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.019+0000 7f14817fa700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f147402ff60 0x7f1474032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:22.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.019+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1484045ad0 con 0x7f14840f5410 2024-01-21T11:19:22.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.019+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1478041010 con 0x7f14840f5410 2024-01-21T11:19:22.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.019+0000 7f1483fff700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f147402ff60 0x7f1474032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:22.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.020+0000 7f1483fff700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f147402ff60 0x7f1474032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f14700096a0 tx=0x7f1470006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:22.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.024+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1478012920 con 0x7f14840f5410 2024-01-21T11:19:22.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.231+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f148410a2a0 con 0x7f14840f5410 2024-01-21T11:19:22.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.232+0000 7f14817fa700 1 -- 172.21.15.178:0/2016174326 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f147802b020 con 0x7f14840f5410 2024-01-21T11:19:22.232 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:22.232 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:22.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f147402ff60 msgr2=0x7f1474032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f147402ff60 0x7f1474032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f14700096a0 tx=0x7f1470006b40).stop 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 msgr2=0x7f14841093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14841093d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1478013510 tx=0x7f147800b040).stop 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 shutdown_connections 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f147402ff60 0x7f1474032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 --2- 172.21.15.178:0/2016174326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14840f5410 0x7f14841093d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 >> 172.21.15.178:0/2016174326 conn(0x7f14840f0b70 msgr2=0x7f14840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 shutdown_connections 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:22.233+0000 7f148b242700 1 -- 172.21.15.178:0/2016174326 wait complete. 2024-01-21T11:19:22.234 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:23.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:21.619139+0000 mgr.smithi089.ufybqw (mgr.14162) 187 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T11:19:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:19:21.624333+0000 mgr.smithi089.ufybqw (mgr.14162) 188 : cephadm [INF] Deploying daemon grafana.smithi089 on smithi089 2024-01-21T11:19:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:21.695663+0000 mgr.smithi089.ufybqw (mgr.14162) 189 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:22.232462+0000 mon.smithi089 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.178:0/2016174326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:23.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:23.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 -- 172.21.15.178:0/1718325529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 msgr2=0x7fc3280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 --2- 172.21.15.178:0/1718325529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3280f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fc318009fb0 tx=0x7fc318009b30).stop 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 -- 172.21.15.178:0/1718325529 shutdown_connections 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 --2- 172.21.15.178:0/1718325529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3280f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 -- 172.21.15.178:0/1718325529 >> 172.21.15.178:0/1718325529 conn(0x7fc3280f09a0 msgr2=0x7fc3280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 -- 172.21.15.178:0/1718325529 shutdown_connections 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 -- 172.21.15.178:0/1718325529 wait complete. 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 Processor -- start 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.735+0000 7fc32759e700 1 -- start start 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32759e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3281064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32759e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc328106990 con 0x7fc3280f51a0 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3281064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3281064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:59306/0 (socket says 172.21.15.178:59306) 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32659c700 1 -- 172.21.15.178:0/3001154365 learned_addr learned my addr 172.21.15.178:0/3001154365 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32659c700 1 -- 172.21.15.178:0/3001154365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc318005040 con 0x7fc3280f51a0 2024-01-21T11:19:24.738 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32659c700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3281064a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc318013510 tx=0x7fc31800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc318009320 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc328106b90 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc318013f00 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.736+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc318014dc0 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.737+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc328107020 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.737+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3180095f0 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.737+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc328045ad0 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.737+0000 7fc316ffd700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc31002ff60 0x7fc310032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.737+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3180410c0 con 0x7fc3280f51a0 2024-01-21T11:19:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.738+0000 7fc325d9b700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc31002ff60 0x7fc310032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:24.740 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.738+0000 7fc325d9b700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc31002ff60 0x7fc310032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc31c0096a0 tx=0x7fc31c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:24.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.742+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc318014420 con 0x7fc3280f51a0 2024-01-21T11:19:24.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.947+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc328107ed0 con 0x7fc3280f51a0 2024-01-21T11:19:24.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.948+0000 7fc316ffd700 1 -- 172.21.15.178:0/3001154365 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc318010230 con 0x7fc3280f51a0 2024-01-21T11:19:24.948 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:24.948 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc31002ff60 msgr2=0x7fc310032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc31002ff60 0x7fc310032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc31c0096a0 tx=0x7fc31c006b40).stop 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 msgr2=0x7fc3281064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3281064a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc318013510 tx=0x7fc31800b040).stop 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 shutdown_connections 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc31002ff60 0x7fc310032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 --2- 172.21.15.178:0/3001154365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280f51a0 0x7fc3281064a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 >> 172.21.15.178:0/3001154365 conn(0x7fc3280f09a0 msgr2=0x7fc32805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 shutdown_connections 2024-01-21T11:19:24.950 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:24.950+0000 7fc32759e700 1 -- 172.21.15.178:0/3001154365 wait complete. 2024-01-21T11:19:24.951 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:25.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:23.696018+0000 mgr.smithi089.ufybqw (mgr.14162) 190 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:26.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:25 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:24.948828+0000 mon.smithi089 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.178:0/3001154365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:26.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:26.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:27.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:25.696402+0000 mgr.smithi089.ufybqw (mgr.14162) 191 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:27.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 -- 172.21.15.178:0/1647139915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 msgr2=0x7f23440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 --2- 172.21.15.178:0/1647139915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f23440f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2334009fb0 tx=0x7f2334009b30).stop 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 -- 172.21.15.178:0/1647139915 shutdown_connections 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 --2- 172.21.15.178:0/1647139915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f23440f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 -- 172.21.15.178:0/1647139915 >> 172.21.15.178:0/1647139915 conn(0x7f23440f0b70 msgr2=0x7f23440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 -- 172.21.15.178:0/1647139915 shutdown_connections 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.482+0000 7f234b8d0700 1 -- 172.21.15.178:0/1647139915 wait complete. 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234b8d0700 1 Processor -- start 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234b8d0700 1 -- start start 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234b8d0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f2344106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234b8d0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2344106c20 con 0x7f23440f5410 2024-01-21T11:19:27.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234966c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f2344106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234966c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f2344106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:59324/0 (socket says 172.21.15.178:59324) 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.483+0000 7f234966c700 1 -- 172.21.15.178:0/3570543760 learned_addr learned my addr 172.21.15.178:0/3570543760 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f234966c700 1 -- 172.21.15.178:0/3570543760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2334005040 con 0x7f23440f5410 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f234966c700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f2344106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f2334014000 tx=0x7f233400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23340095f0 con 0x7f23440f5410 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2344106e20 con 0x7f23440f5410 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2334007e70 con 0x7f23440f5410 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2334014ea0 con 0x7f23440f5410 2024-01-21T11:19:27.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.484+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23441072b0 con 0x7f23440f5410 2024-01-21T11:19:27.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.485+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2334025070 con 0x7f23440f5410 2024-01-21T11:19:27.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.485+0000 7f2339ffb700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f233002ff60 0x7f2330032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:27.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.485+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2334040970 con 0x7f23440f5410 2024-01-21T11:19:27.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.485+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2344045ad0 con 0x7f23440f5410 2024-01-21T11:19:27.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.485+0000 7f2348e6b700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f233002ff60 0x7f2330032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:27.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.486+0000 7f2348e6b700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f233002ff60 0x7f2330032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f23400096a0 tx=0x7f2340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:27.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.490+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2334026350 con 0x7f23440f5410 2024-01-21T11:19:27.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.701+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2344107fd0 con 0x7f23440f5410 2024-01-21T11:19:27.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.701+0000 7f2339ffb700 1 -- 172.21.15.178:0/3570543760 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2334012330 con 0x7f23440f5410 2024-01-21T11:19:27.702 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:27.702 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:27.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f233002ff60 msgr2=0x7f2330032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:27.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f233002ff60 0x7f2330032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f23400096a0 tx=0x7f2340006b40).stop 2024-01-21T11:19:27.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 msgr2=0x7f2344106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:27.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f2344106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f2334014000 tx=0x7f233400b040).stop 2024-01-21T11:19:27.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 shutdown_connections 2024-01-21T11:19:27.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f233002ff60 0x7f2330032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:27.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 --2- 172.21.15.178:0/3570543760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23440f5410 0x7f2344106730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:27.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 >> 172.21.15.178:0/3570543760 conn(0x7f23440f0b70 msgr2=0x7f23440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:27.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 shutdown_connections 2024-01-21T11:19:27.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:27.703+0000 7f234b8d0700 1 -- 172.21.15.178:0/3570543760 wait complete. 2024-01-21T11:19:27.704 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:28.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:27 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:27.702149+0000 mon.smithi089 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.178:0/3570543760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:29.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:29.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:29.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:27.696734+0000 mgr.smithi089.ufybqw (mgr.14162) 192 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:30.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 -- 172.21.15.178:0/711895577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 msgr2=0x7f00200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:30.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 --2- 172.21.15.178:0/711895577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00200f55c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f0008009fb0 tx=0x7f0008009b30).stop 2024-01-21T11:19:30.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 -- 172.21.15.178:0/711895577 shutdown_connections 2024-01-21T11:19:30.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 --2- 172.21.15.178:0/711895577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00200f55c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 -- 172.21.15.178:0/711895577 >> 172.21.15.178:0/711895577 conn(0x7f00200f09a0 msgr2=0x7f00200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 -- 172.21.15.178:0/711895577 shutdown_connections 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 -- 172.21.15.178:0/711895577 wait complete. 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 Processor -- start 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 -- start start 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.265+0000 7f0024ae4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00201091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f0024ae4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0020060220 con 0x7f00200f51a0 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f001f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00201091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f001f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00201091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:59348/0 (socket says 172.21.15.178:59348) 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f001f7fe700 1 -- 172.21.15.178:0/2761514695 learned_addr learned my addr 172.21.15.178:0/2761514695 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f001f7fe700 1 -- 172.21.15.178:0/2761514695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008005040 con 0x7f00200f51a0 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f001f7fe700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00201091d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f0008013510 tx=0x7f000800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0008009320 con 0x7f00200f51a0 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0020060420 con 0x7f00200f51a0 2024-01-21T11:19:30.267 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0008013f00 con 0x7f00200f51a0 2024-01-21T11:19:30.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0008014dc0 con 0x7f00200f51a0 2024-01-21T11:19:30.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.266+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0020109880 con 0x7f00200f51a0 2024-01-21T11:19:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.267+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0008014420 con 0x7f00200f51a0 2024-01-21T11:19:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.267+0000 7f0017fff700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f000c02ff60 0x7f000c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.267+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0008041050 con 0x7f00200f51a0 2024-01-21T11:19:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.267+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0020059100 con 0x7f00200f51a0 2024-01-21T11:19:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.267+0000 7f001effd700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f000c02ff60 0x7f000c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.268+0000 7f001effd700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f000c02ff60 0x7f000c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f00100096a0 tx=0x7f0010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:30.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.272+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00080095f0 con 0x7f00200f51a0 2024-01-21T11:19:30.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.477+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00200fabe0 con 0x7f00200f51a0 2024-01-21T11:19:30.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.478+0000 7f0017fff700 1 -- 172.21.15.178:0/2761514695 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f000802b020 con 0x7f00200f51a0 2024-01-21T11:19:30.478 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:30.478 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f000c02ff60 msgr2=0x7f000c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f000c02ff60 0x7f000c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f00100096a0 tx=0x7f0010006b40).stop 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 msgr2=0x7f00201091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00201091d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f0008013510 tx=0x7f000800b040).stop 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 shutdown_connections 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f000c02ff60 0x7f000c032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.479+0000 7f0024ae4700 1 --2- 172.21.15.178:0/2761514695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00200f51a0 0x7f00201091d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.480+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 >> 172.21.15.178:0/2761514695 conn(0x7f00200f09a0 msgr2=0x7f00200f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.480+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 shutdown_connections 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:30.480+0000 7f0024ae4700 1 -- 172.21.15.178:0/2761514695 wait complete. 2024-01-21T11:19:30.480 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:31.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:29.696959+0000 mgr.smithi089.ufybqw (mgr.14162) 193 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:31.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:30.478570+0000 mon.smithi089 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.178:0/2761514695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:31.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:31.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:33.036 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 -- 172.21.15.178:0/508264464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 msgr2=0x7f0ac00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:33.036 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 --2- 172.21.15.178:0/508264464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f0ab0009fb0 tx=0x7f0ab0009b30).stop 2024-01-21T11:19:33.036 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 -- 172.21.15.178:0/508264464 shutdown_connections 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 --2- 172.21.15.178:0/508264464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 -- 172.21.15.178:0/508264464 >> 172.21.15.178:0/508264464 conn(0x7f0ac00f0b70 msgr2=0x7f0ac00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 -- 172.21.15.178:0/508264464 shutdown_connections 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 -- 172.21.15.178:0/508264464 wait complete. 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 Processor -- start 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.033+0000 7f0ac69fd700 1 -- start start 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0ac69fd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:33.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0ac69fd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00fb820 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57358/0 (socket says 172.21.15.178:57358) 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abffff700 1 -- 172.21.15.178:0/3642606323 learned_addr learned my addr 172.21.15.178:0/3642606323 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abffff700 1 -- 172.21.15.178:0/3642606323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0005040 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abffff700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00fd3a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f0ab0014000 tx=0x7f0ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab00095f0 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab0007e70 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac00fb9c0 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab0014ea0 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.034+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00fbe50 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.035+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ab0025070 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.035+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0059100 con 0x7f0ac00f5410 2024-01-21T11:19:33.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.035+0000 7f0abcff9700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0aa802ffb0 0x7f0aa8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:33.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.035+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ab00409e0 con 0x7f0ac00f5410 2024-01-21T11:19:33.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.035+0000 7f0abf7fe700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0aa802ffb0 0x7f0aa8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:33.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.036+0000 7f0abf7fe700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0aa802ffb0 0x7f0aa8032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0ab40096a0 tx=0x7f0ab4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:33.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.040+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ab000f4e0 con 0x7f0ac00f5410 2024-01-21T11:19:33.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:31.697165+0000 mgr.smithi089.ufybqw (mgr.14162) 194 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:33.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.248+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ac00fae60 con 0x7f0ac00f5410 2024-01-21T11:19:33.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.249+0000 7f0abcff9700 1 -- 172.21.15.178:0/3642606323 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ab000f300 con 0x7f0ac00f5410 2024-01-21T11:19:33.249 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:33.249 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0aa802ffb0 msgr2=0x7f0aa8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0aa802ffb0 0x7f0aa8032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0ab40096a0 tx=0x7f0ab4006b40).stop 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 msgr2=0x7f0ac00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00fd3a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f0ab0014000 tx=0x7f0ab000b040).stop 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 shutdown_connections 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0aa802ffb0 0x7f0aa8032470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:33.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 --2- 172.21.15.178:0/3642606323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ac00f5410 0x7f0ac00fd3a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:33.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 >> 172.21.15.178:0/3642606323 conn(0x7f0ac00f0b70 msgr2=0x7f0ac005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:33.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.250+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 shutdown_connections 2024-01-21T11:19:33.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:33.251+0000 7f0ac69fd700 1 -- 172.21.15.178:0/3642606323 wait complete. 2024-01-21T11:19:33.251 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:34.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:33 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:33.249585+0000 mon.smithi089 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.178:0/3642606323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:34.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:34.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:35.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:33.697376+0000 mgr.smithi089.ufybqw (mgr.14162) 195 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:35.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 -- 172.21.15.178:0/3294570935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 msgr2=0x7fea4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:35.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 --2- 172.21.15.178:0/3294570935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fea40009fb0 tx=0x7fea40009b30).stop 2024-01-21T11:19:35.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 -- 172.21.15.178:0/3294570935 shutdown_connections 2024-01-21T11:19:35.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 --2- 172.21.15.178:0/3294570935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:35.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 -- 172.21.15.178:0/3294570935 >> 172.21.15.178:0/3294570935 conn(0x7fea4c0f0b70 msgr2=0x7fea4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:35.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 -- 172.21.15.178:0/3294570935 shutdown_connections 2024-01-21T11:19:35.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 -- 172.21.15.178:0/3294570935 wait complete. 2024-01-21T11:19:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 Processor -- start 2024-01-21T11:19:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.799+0000 7fea50849700 1 -- start start 2024-01-21T11:19:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea50849700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea50849700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c105b20 con 0x7fea4c0f5410 2024-01-21T11:19:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea4a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea4a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57364/0 (socket says 172.21.15.178:57364) 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea4a59c700 1 -- 172.21.15.178:0/3223345122 learned_addr learned my addr 172.21.15.178:0/3223345122 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea4a59c700 1 -- 172.21.15.178:0/3223345122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea40005040 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea4a59c700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fea40013510 tx=0x7fea4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea40009320 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.800+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c105d20 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea40013f00 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea40014d90 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c1061b0 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea40014420 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c045ad0 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea3affd700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea3402ff60 0x7fea34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.801+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea40041110 con 0x7fea4c0f5410 2024-01-21T11:19:35.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.802+0000 7fea49d9b700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea3402ff60 0x7fea34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:35.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.802+0000 7fea49d9b700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea3402ff60 0x7fea34032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fea3c0096a0 tx=0x7fea3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:35.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:35.806+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea400095f0 con 0x7fea4c0f5410 2024-01-21T11:19:36.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.012+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea4c0fae60 con 0x7fea4c0f5410 2024-01-21T11:19:36.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.012+0000 7fea3affd700 1 -- 172.21.15.178:0/3223345122 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea4001db70 con 0x7fea4c0f5410 2024-01-21T11:19:36.012 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:36.013 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea3402ff60 msgr2=0x7fea34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea3402ff60 0x7fea34032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fea3c0096a0 tx=0x7fea3c006b40).stop 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 msgr2=0x7fea4c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fea40013510 tx=0x7fea4000b040).stop 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 shutdown_connections 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea3402ff60 0x7fea34032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 --2- 172.21.15.178:0/3223345122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea4c0f5410 0x7fea4c105630 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 >> 172.21.15.178:0/3223345122 conn(0x7fea4c0f0b70 msgr2=0x7fea4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:36.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 shutdown_connections 2024-01-21T11:19:36.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:36.014+0000 7fea50849700 1 -- 172.21.15.178:0/3223345122 wait complete. 2024-01-21T11:19:36.015 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:37.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:35.697625+0000 mgr.smithi089.ufybqw (mgr.14162) 196 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:37.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:36.013200+0000 mon.smithi089 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.178:0/3223345122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:37.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:37.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 -- 172.21.15.178:0/1322721625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 msgr2=0x7f36e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 --2- 172.21.15.178:0/1322721625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e80f55c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f36d8009fb0 tx=0x7f36d8009b30).stop 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 -- 172.21.15.178:0/1322721625 shutdown_connections 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 --2- 172.21.15.178:0/1322721625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e80f55c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 -- 172.21.15.178:0/1322721625 >> 172.21.15.178:0/1322721625 conn(0x7f36e80f09a0 msgr2=0x7f36e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 -- 172.21.15.178:0/1322721625 shutdown_connections 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.525+0000 7f36ecf80700 1 -- 172.21.15.178:0/1322721625 wait complete. 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36ecf80700 1 Processor -- start 2024-01-21T11:19:38.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36ecf80700 1 -- start start 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36ecf80700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e81091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36ecf80700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8060240 con 0x7f36e80f51a0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36e77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e81091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36e77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e81091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57376/0 (socket says 172.21.15.178:57376) 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.526+0000 7f36e77fe700 1 -- 172.21.15.178:0/3627605964 learned_addr learned my addr 172.21.15.178:0/3627605964 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36e77fe700 1 -- 172.21.15.178:0/3627605964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8005040 con 0x7f36e80f51a0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36e77fe700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e81091e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f36d8013510 tx=0x7f36d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d8009320 con 0x7f36e80f51a0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e8060440 con 0x7f36e80f51a0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36d8013f00 con 0x7f36e80f51a0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d8014dc0 con 0x7f36e80f51a0 2024-01-21T11:19:38.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.527+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e8109890 con 0x7f36e80f51a0 2024-01-21T11:19:38.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.528+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36d8014420 con 0x7f36e80f51a0 2024-01-21T11:19:38.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.528+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e8045ad0 con 0x7f36e80f51a0 2024-01-21T11:19:38.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.528+0000 7f36cffff700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36d002ff60 0x7f36d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:38.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.528+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36d8041110 con 0x7f36e80f51a0 2024-01-21T11:19:38.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.528+0000 7f36e6ffd700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36d002ff60 0x7f36d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:38.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.528+0000 7f36e6ffd700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36d002ff60 0x7f36d0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f36dc0096a0 tx=0x7f36dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:38.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.532+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36d8026bf0 con 0x7f36e80f51a0 2024-01-21T11:19:38.739 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.740+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36e810a0b0 con 0x7f36e80f51a0 2024-01-21T11:19:38.740 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.740+0000 7f36cffff700 1 -- 172.21.15.178:0/3627605964 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36d801cb10 con 0x7f36e80f51a0 2024-01-21T11:19:38.740 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:38.740 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36d002ff60 msgr2=0x7f36d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36d002ff60 0x7f36d0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f36dc0096a0 tx=0x7f36dc006b40).stop 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 msgr2=0x7f36e81091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e81091e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f36d8013510 tx=0x7f36d800b040).stop 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 shutdown_connections 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f36d002ff60 0x7f36d0032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 --2- 172.21.15.178:0/3627605964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36e80f51a0 0x7f36e81091e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 >> 172.21.15.178:0/3627605964 conn(0x7f36e80f09a0 msgr2=0x7f36e80f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:38.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 shutdown_connections 2024-01-21T11:19:38.743 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:38.742+0000 7f36ecf80700 1 -- 172.21.15.178:0/3627605964 wait complete. 2024-01-21T11:19:38.743 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:38.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:37.477215+0000 mon.smithi089 (mon.0) 263 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' 2024-01-21T11:19:38.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:19:37.478471+0000 mgr.smithi089.ufybqw (mgr.14162) 197 : cephadm [INF] Deploying daemon node-exporter.smithi089 on smithi089 2024-01-21T11:19:38.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:37.697878+0000 mgr.smithi089.ufybqw (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:39.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:38.741312+0000 mon.smithi089 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.178:0/3627605964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:40.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:40.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:41.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:39.698194+0000 mgr.smithi089.ufybqw (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:41.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 -- 172.21.15.178:0/572392500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 msgr2=0x7f30040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 --2- 172.21.15.178:0/572392500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f30040f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f2ff4009fb0 tx=0x7f2ff4009b30).stop 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 -- 172.21.15.178:0/572392500 shutdown_connections 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 --2- 172.21.15.178:0/572392500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f30040f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 -- 172.21.15.178:0/572392500 >> 172.21.15.178:0/572392500 conn(0x7f30040f1c70 msgr2=0x7f30040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 -- 172.21.15.178:0/572392500 shutdown_connections 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 -- 172.21.15.178:0/572392500 wait complete. 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 Processor -- start 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.312+0000 7f300a56b700 1 -- start start 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f300a56b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f3004106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f300a56b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004106c20 con 0x7f30040f6510 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f3003fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f3004106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:41.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f3003fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f3004106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57392/0 (socket says 172.21.15.178:57392) 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f3003fff700 1 -- 172.21.15.178:0/1350585196 learned_addr learned my addr 172.21.15.178:0/1350585196 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f3003fff700 1 -- 172.21.15.178:0/1350585196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4005040 con 0x7f30040f6510 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f3003fff700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f3004106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2ff4013510 tx=0x7f2ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.313+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff4009320 con 0x7f30040f6510 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.314+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3004106e20 con 0x7f30040f6510 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.314+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ff4013f00 con 0x7f30040f6510 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.314+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff4014d90 con 0x7f30040f6510 2024-01-21T11:19:41.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.314+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30041072b0 con 0x7f30040f6510 2024-01-21T11:19:41.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.314+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ff40263a0 con 0x7f30040f6510 2024-01-21T11:19:41.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.314+0000 7f3000ff9700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2fec02ff60 0x7f2fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:41.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.315+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ff4040fe0 con 0x7f30040f6510 2024-01-21T11:19:41.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.315+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3004045ad0 con 0x7f30040f6510 2024-01-21T11:19:41.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.315+0000 7f30037fe700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2fec02ff60 0x7f2fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:41.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.315+0000 7f30037fe700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2fec02ff60 0x7f2fec032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f2ff80096a0 tx=0x7f2ff8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:41.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.320+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ff4012920 con 0x7f30040f6510 2024-01-21T11:19:41.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.525+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30040fbf60 con 0x7f30040f6510 2024-01-21T11:19:41.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.526+0000 7f3000ff9700 1 -- 172.21.15.178:0/1350585196 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ff402b020 con 0x7f30040f6510 2024-01-21T11:19:41.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:41.526 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:41.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2fec02ff60 msgr2=0x7f2fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2fec02ff60 0x7f2fec032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f2ff80096a0 tx=0x7f2ff8006b40).stop 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 msgr2=0x7f3004106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f3004106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2ff4013510 tx=0x7f2ff400b040).stop 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 shutdown_connections 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2fec02ff60 0x7f2fec032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 --2- 172.21.15.178:0/1350585196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30040f6510 0x7f3004106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.527+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 >> 172.21.15.178:0/1350585196 conn(0x7f30040f1c70 msgr2=0x7f30040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.528+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 shutdown_connections 2024-01-21T11:19:41.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:41.528+0000 7f300a56b700 1 -- 172.21.15.178:0/1350585196 wait complete. 2024-01-21T11:19:41.529 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:42.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:41 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:41.526669+0000 mon.smithi089 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.178:0/1350585196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:42.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:42.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:43.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:41.698588+0000 mgr.smithi089.ufybqw (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 -- 172.21.15.178:0/3533244039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 msgr2=0x7fea700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 --2- 172.21.15.178:0/3533244039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea700f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fea60009fb0 tx=0x7fea60009b30).stop 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 -- 172.21.15.178:0/3533244039 shutdown_connections 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 --2- 172.21.15.178:0/3533244039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea700f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 -- 172.21.15.178:0/3533244039 >> 172.21.15.178:0/3533244039 conn(0x7fea700f0b70 msgr2=0x7fea700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 -- 172.21.15.178:0/3533244039 shutdown_connections 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.062+0000 7fea77ea8700 1 -- 172.21.15.178:0/3533244039 wait complete. 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea77ea8700 1 Processor -- start 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea77ea8700 1 -- start start 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea77ea8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea701066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea77ea8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea70106bd0 con 0x7fea700f5410 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea75c44700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea701066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea75c44700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea701066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47362/0 (socket says 172.21.15.178:47362) 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea75c44700 1 -- 172.21.15.178:0/462695932 learned_addr learned my addr 172.21.15.178:0/462695932 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.063+0000 7fea75c44700 1 -- 172.21.15.178:0/462695932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60005040 con 0x7fea700f5410 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.064+0000 7fea75c44700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea701066e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fea60013510 tx=0x7fea6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.064+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea60009320 con 0x7fea700f5410 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.064+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea70106dd0 con 0x7fea700f5410 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.064+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea60013f00 con 0x7fea700f5410 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.064+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea60014d90 con 0x7fea700f5410 2024-01-21T11:19:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.064+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea70107260 con 0x7fea700f5410 2024-01-21T11:19:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.065+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea70045ad0 con 0x7fea700f5410 2024-01-21T11:19:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.065+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea600095f0 con 0x7fea700f5410 2024-01-21T11:19:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.065+0000 7fea667fc700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea5c02ff60 0x7fea5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.065+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea60040fc0 con 0x7fea700f5410 2024-01-21T11:19:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.065+0000 7fea75443700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea5c02ff60 0x7fea5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.065+0000 7fea75443700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea5c02ff60 0x7fea5c032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fea6c0096a0 tx=0x7fea6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.070+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea60014420 con 0x7fea700f5410 2024-01-21T11:19:44.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.279+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea700fae60 con 0x7fea700f5410 2024-01-21T11:19:44.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.280+0000 7fea667fc700 1 -- 172.21.15.178:0/462695932 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea60010230 con 0x7fea700f5410 2024-01-21T11:19:44.281 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:44.281 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:44.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.281+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea5c02ff60 msgr2=0x7fea5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:44.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.281+0000 7fea77ea8700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea5c02ff60 0x7fea5c032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fea6c0096a0 tx=0x7fea6c006b40).stop 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.281+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 msgr2=0x7fea701066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.281+0000 7fea77ea8700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea701066e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fea60013510 tx=0x7fea6000b040).stop 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.282+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 shutdown_connections 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.282+0000 7fea77ea8700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fea5c02ff60 0x7fea5c032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.282+0000 7fea77ea8700 1 --2- 172.21.15.178:0/462695932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700f5410 0x7fea701066e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.282+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 >> 172.21.15.178:0/462695932 conn(0x7fea700f0b70 msgr2=0x7fea7005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.282+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 shutdown_connections 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:44.282+0000 7fea77ea8700 1 -- 172.21.15.178:0/462695932 wait complete. 2024-01-21T11:19:44.283 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:45.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:43.698900+0000 mgr.smithi089.ufybqw (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:45.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:44.281768+0000 mon.smithi089 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.178:0/462695932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:45.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:45.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:46.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- 172.21.15.178:0/449338878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 msgr2=0x7f59340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:46.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 --2- 172.21.15.178:0/449338878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f59340f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f5930009fb0 tx=0x7f5930009b30).stop 2024-01-21T11:19:46.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- 172.21.15.178:0/449338878 shutdown_connections 2024-01-21T11:19:46.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 --2- 172.21.15.178:0/449338878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f59340f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:46.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- 172.21.15.178:0/449338878 >> 172.21.15.178:0/449338878 conn(0x7f59340f0b70 msgr2=0x7f59340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- 172.21.15.178:0/449338878 shutdown_connections 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- 172.21.15.178:0/449338878 wait complete. 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 Processor -- start 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- start start 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f5934105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.813+0000 7f593b8e0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5934105b20 con 0x7f59340f5410 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593967c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f5934105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593967c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f5934105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47376/0 (socket says 172.21.15.178:47376) 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593967c700 1 -- 172.21.15.178:0/3215513858 learned_addr learned my addr 172.21.15.178:0/3215513858 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593967c700 1 -- 172.21.15.178:0/3215513858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930005040 con 0x7f59340f5410 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593967c700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f5934105630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5930013510 tx=0x7f593000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5930009320 con 0x7f59340f5410 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5934105d20 con 0x7f59340f5410 2024-01-21T11:19:46.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5930013f00 con 0x7f59340f5410 2024-01-21T11:19:46.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5930014d90 con 0x7f59340f5410 2024-01-21T11:19:46.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.814+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59341061b0 con 0x7f59340f5410 2024-01-21T11:19:46.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.815+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5934045ad0 con 0x7f59340f5410 2024-01-21T11:19:46.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.815+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59300095f0 con 0x7f59340f5410 2024-01-21T11:19:46.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.815+0000 7f5929ffb700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f592002ff60 0x7f5920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:46.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.815+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59300410c0 con 0x7f59340f5410 2024-01-21T11:19:46.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.816+0000 7f5938e7b700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f592002ff60 0x7f5920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:46.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.816+0000 7f5938e7b700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f592002ff60 0x7f5920032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:46.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:46.820+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f593001d7f0 con 0x7f59340f5410 2024-01-21T11:19:47.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.027+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59340fae60 con 0x7f59340f5410 2024-01-21T11:19:47.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.028+0000 7f5929ffb700 1 -- 172.21.15.178:0/3215513858 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5930010230 con 0x7f59340f5410 2024-01-21T11:19:47.029 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:47.029 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f592002ff60 msgr2=0x7f5920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f592002ff60 0x7f5920032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).stop 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 msgr2=0x7f5934105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f5934105630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5930013510 tx=0x7f593000b040).stop 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 shutdown_connections 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f592002ff60 0x7f5920032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 --2- 172.21.15.178:0/3215513858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59340f5410 0x7f5934105630 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 >> 172.21.15.178:0/3215513858 conn(0x7f59340f0b70 msgr2=0x7f59340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 shutdown_connections 2024-01-21T11:19:47.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:47.030+0000 7f593b8e0700 1 -- 172.21.15.178:0/3215513858 wait complete. 2024-01-21T11:19:47.032 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:47.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:45.699230+0000 mgr.smithi089.ufybqw (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:48.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:47.029970+0000 mon.smithi089 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.178:0/3215513858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:48.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:48.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:49.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:47.699613+0000 mgr.smithi089.ufybqw (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.555+0000 7f8a26e86700 1 -- 172.21.15.178:0/3928005561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 msgr2=0x7f8a200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.555+0000 7f8a26e86700 1 --2- 172.21.15.178:0/3928005561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a200f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8a14009fb0 tx=0x7f8a14009b30).stop 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 -- 172.21.15.178:0/3928005561 shutdown_connections 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 --2- 172.21.15.178:0/3928005561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a200f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 -- 172.21.15.178:0/3928005561 >> 172.21.15.178:0/3928005561 conn(0x7f8a200f0b70 msgr2=0x7f8a200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 -- 172.21.15.178:0/3928005561 shutdown_connections 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 -- 172.21.15.178:0/3928005561 wait complete. 2024-01-21T11:19:49.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 Processor -- start 2024-01-21T11:19:49.558 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.556+0000 7f8a26e86700 1 -- start start 2024-01-21T11:19:49.559 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a26e86700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a20105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:49.559 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a26e86700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a20105af0 con 0x7f8a200f5410 2024-01-21T11:19:49.559 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a24c22700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a20105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a24c22700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a20105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47400/0 (socket says 172.21.15.178:47400) 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a24c22700 1 -- 172.21.15.178:0/1032651776 learned_addr learned my addr 172.21.15.178:0/1032651776 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a24c22700 1 -- 172.21.15.178:0/1032651776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a14005040 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.557+0000 7f8a24c22700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a20105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8a14014000 tx=0x7f8a1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a140095f0 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a14007e70 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a20105cf0 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a14014ea0 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a20106180 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a14025070 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.558+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a20045ad0 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.559+0000 7f8a1d7fa700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8a1002ff60 0x7f8a10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.559+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a140409f0 con 0x7f8a200f5410 2024-01-21T11:19:49.560 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.559+0000 7f8a1ffff700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8a1002ff60 0x7f8a10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:49.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.559+0000 7f8a1ffff700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8a1002ff60 0x7f8a10032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8a0c0096a0 tx=0x7f8a0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:49.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.563+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a1400f4e0 con 0x7f8a200f5410 2024-01-21T11:19:49.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.768+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a200fae60 con 0x7f8a200f5410 2024-01-21T11:19:49.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.769+0000 7f8a1d7fa700 1 -- 172.21.15.178:0/1032651776 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a1400f300 con 0x7f8a200f5410 2024-01-21T11:19:49.770 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:49.770 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8a1002ff60 msgr2=0x7f8a10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8a1002ff60 0x7f8a10032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8a0c0096a0 tx=0x7f8a0c006b40).stop 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 msgr2=0x7f8a20105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a20105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8a14014000 tx=0x7f8a1400b040).stop 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 shutdown_connections 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8a1002ff60 0x7f8a10032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 --2- 172.21.15.178:0/1032651776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200f5410 0x7f8a20105600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:49.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 >> 172.21.15.178:0/1032651776 conn(0x7f8a200f0b70 msgr2=0x7f8a200f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:49.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 shutdown_connections 2024-01-21T11:19:49.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:49.771+0000 7f8a26e86700 1 -- 172.21.15.178:0/1032651776 wait complete. 2024-01-21T11:19:49.773 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:51.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:51.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:51.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:49.699857+0000 mgr.smithi089.ufybqw (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:51.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:49.770964+0000 mon.smithi089 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.178:0/1032651776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 -- 172.21.15.178:0/403963856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e80f5410 msgr2=0x7f47e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 --2- 172.21.15.178:0/403963856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e80f5410 0x7f47e80f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f47d8009fb0 tx=0x7f47d8009b30).stop 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 -- 172.21.15.178:0/403963856 shutdown_connections 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 --2- 172.21.15.178:0/403963856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e80f5410 0x7f47e80f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 -- 172.21.15.178:0/403963856 >> 172.21.15.178:0/403963856 conn(0x7f47e80f0b70 msgr2=0x7f47e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 -- 172.21.15.178:0/403963856 shutdown_connections 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.328+0000 7f47efbef700 1 -- 172.21.15.178:0/403963856 wait complete. 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47efbef700 1 Processor -- start 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47efbef700 1 -- start start 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47efbef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 0x7f47e8105b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:52.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47efbef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e8106050 con 0x7f47e8105740 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47ed98b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 0x7f47e8105b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47ed98b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 0x7f47e8105b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:45976/0 (socket says 172.21.15.178:45976) 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47ed98b700 1 -- 172.21.15.178:0/1957755945 learned_addr learned my addr 172.21.15.178:0/1957755945 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.329+0000 7f47ed98b700 1 -- 172.21.15.178:0/1957755945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d8005040 con 0x7f47e8105740 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.330+0000 7f47ed98b700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 0x7f47e8105b60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f47d8009f80 tx=0x7f47d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.330+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47d80136c0 con 0x7f47e8105740 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.330+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e8106250 con 0x7f47e8105740 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.330+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47d8013e70 con 0x7f47e8105740 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.330+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47d8014420 con 0x7f47e8105740 2024-01-21T11:19:52.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.330+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e8106da0 con 0x7f47e8105740 2024-01-21T11:19:52.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.331+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47d8025060 con 0x7f47e8105740 2024-01-21T11:19:52.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.331+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e8059100 con 0x7f47e8105740 2024-01-21T11:19:52.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.331+0000 7f47de7fc700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f47d402ff60 0x7f47d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:52.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.331+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47d8040e00 con 0x7f47e8105740 2024-01-21T11:19:52.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.331+0000 7f47ed18a700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f47d402ff60 0x7f47d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:52.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.331+0000 7f47ed18a700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f47d402ff60 0x7f47d4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f47e40096a0 tx=0x7f47e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:52.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.336+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47d801d3a0 con 0x7f47e8105740 2024-01-21T11:19:52.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.542+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47e8045ad0 con 0x7f47e8105740 2024-01-21T11:19:52.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.543+0000 7f47de7fc700 1 -- 172.21.15.178:0/1957755945 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47d802d2d0 con 0x7f47e8105740 2024-01-21T11:19:52.544 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:52.544 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:52.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f47d402ff60 msgr2=0x7f47d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:52.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f47d402ff60 0x7f47d4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f47e40096a0 tx=0x7f47e4006b40).stop 2024-01-21T11:19:52.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 msgr2=0x7f47e8105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:52.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 0x7f47e8105b60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f47d8009f80 tx=0x7f47d8009b30).stop 2024-01-21T11:19:52.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 shutdown_connections 2024-01-21T11:19:52.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f47d402ff60 0x7f47d4032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:52.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 --2- 172.21.15.178:0/1957755945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e8105740 0x7f47e8105b60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:52.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 >> 172.21.15.178:0/1957755945 conn(0x7f47e80f0b70 msgr2=0x7f47e80f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:52.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 shutdown_connections 2024-01-21T11:19:52.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:52.544+0000 7f47efbef700 1 -- 172.21.15.178:0/1957755945 wait complete. 2024-01-21T11:19:52.546 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:53.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:51.700091+0000 mgr.smithi089.ufybqw (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:53.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:52.544426+0000 mon.smithi089 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.178:0/1957755945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:53.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:53.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:55.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- 172.21.15.178:0/1908675487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 msgr2=0x7f32340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 --2- 172.21.15.178:0/1908675487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32340f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f3224009fb0 tx=0x7f3224009b30).stop 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- 172.21.15.178:0/1908675487 shutdown_connections 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 --2- 172.21.15.178:0/1908675487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32340f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- 172.21.15.178:0/1908675487 >> 172.21.15.178:0/1908675487 conn(0x7f32340f09a0 msgr2=0x7f32340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- 172.21.15.178:0/1908675487 shutdown_connections 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- 172.21.15.178:0/1908675487 wait complete. 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 Processor -- start 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- start start 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32341064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.039+0000 7f323a3c0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3234106990 con 0x7f32340f51a0 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f32393be700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32341064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f32393be700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32341064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:45990/0 (socket says 172.21.15.178:45990) 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f32393be700 1 -- 172.21.15.178:0/1737714381 learned_addr learned my addr 172.21.15.178:0/1737714381 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f32393be700 1 -- 172.21.15.178:0/1737714381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224005040 con 0x7f32340f51a0 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f32393be700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32341064a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3224009830 tx=0x7f322400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224009320 con 0x7f32340f51a0 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3234106b90 con 0x7f32340f51a0 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32240136c0 con 0x7f32340f51a0 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.040+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f322401ee90 con 0x7f32340f51a0 2024-01-21T11:19:55.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.041+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3234107020 con 0x7f32340f51a0 2024-01-21T11:19:55.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.041+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3234045ad0 con 0x7f32340f51a0 2024-01-21T11:19:55.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.042+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f322401e420 con 0x7f32340f51a0 2024-01-21T11:19:55.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.042+0000 7f3229ffb700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f322002ff10 0x7f32200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:55.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.042+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3224040f40 con 0x7f32340f51a0 2024-01-21T11:19:55.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.042+0000 7f3238bbd700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f322002ff10 0x7f32200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:55.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.043+0000 7f3238bbd700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f322002ff10 0x7f32200323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f32300096a0 tx=0x7f3230006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:55.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.047+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f322406d2b0 con 0x7f32340f51a0 2024-01-21T11:19:55.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:53.700459+0000 mgr.smithi089.ufybqw (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:55.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.253+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3234107ed0 con 0x7f32340f51a0 2024-01-21T11:19:55.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.254+0000 7f3229ffb700 1 -- 172.21.15.178:0/1737714381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32240095f0 con 0x7f32340f51a0 2024-01-21T11:19:55.254 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:55.255 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:55.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f322002ff10 msgr2=0x7f32200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:55.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f322002ff10 0x7f32200323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f32300096a0 tx=0x7f3230006b40).stop 2024-01-21T11:19:55.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 msgr2=0x7f32341064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:55.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32341064a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3224009830 tx=0x7f322400b040).stop 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 shutdown_connections 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f322002ff10 0x7f32200323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 --2- 172.21.15.178:0/1737714381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32340f51a0 0x7f32341064a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 >> 172.21.15.178:0/1737714381 conn(0x7f32340f09a0 msgr2=0x7f323405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 shutdown_connections 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:55.255+0000 7f323a3c0700 1 -- 172.21.15.178:0/1737714381 wait complete. 2024-01-21T11:19:55.257 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:56.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:55 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:55.255451+0000 mon.smithi089 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.178:0/1737714381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:56.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:56.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:19:57.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:55.700840+0000 mgr.smithi089.ufybqw (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:57.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- 172.21.15.178:0/2657401820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 msgr2=0x7f74800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:57.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 --2- 172.21.15.178:0/2657401820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f74800f90f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f7470009fb0 tx=0x7f7470009b30).stop 2024-01-21T11:19:57.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- 172.21.15.178:0/2657401820 shutdown_connections 2024-01-21T11:19:57.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 --2- 172.21.15.178:0/2657401820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f74800f90f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:57.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- 172.21.15.178:0/2657401820 >> 172.21.15.178:0/2657401820 conn(0x7f74800f0b70 msgr2=0x7f74800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- 172.21.15.178:0/2657401820 shutdown_connections 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- 172.21.15.178:0/2657401820 wait complete. 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 Processor -- start 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- start start 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f7480106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.852+0000 7f74863c9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7480106b80 con 0x7f7480060280 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f7480106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f7480106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:46004/0 (socket says 172.21.15.178:46004) 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747ffff700 1 -- 172.21.15.178:0/291954111 learned_addr learned my addr 172.21.15.178:0/291954111 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747ffff700 1 -- 172.21.15.178:0/291954111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7470005040 con 0x7f7480060280 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747ffff700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f7480106690 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f7470013510 tx=0x7f747000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7470009320 con 0x7f7480060280 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7480106d80 con 0x7f7480060280 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7470013f00 con 0x7f7480060280 2024-01-21T11:19:57.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7470014dc0 con 0x7f7480060280 2024-01-21T11:19:57.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.853+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7480107210 con 0x7f7480060280 2024-01-21T11:19:57.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.854+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74700095f0 con 0x7f7480060280 2024-01-21T11:19:57.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.854+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7480045ad0 con 0x7f7480060280 2024-01-21T11:19:57.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.854+0000 7f747cff9700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f746802ff60 0x7f7468032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:19:57.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.854+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74700410c0 con 0x7f7480060280 2024-01-21T11:19:57.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.854+0000 7f747f7fe700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f746802ff60 0x7f7468032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:19:57.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.855+0000 7f747f7fe700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f746802ff60 0x7f7468032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f74740096a0 tx=0x7f7474006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:19:57.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:57.859+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f747001c790 con 0x7f7480060280 2024-01-21T11:19:58.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.063+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74801080c0 con 0x7f7480060280 2024-01-21T11:19:58.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.063+0000 7f747cff9700 1 -- 172.21.15.178:0/291954111 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7470010230 con 0x7f7480060280 2024-01-21T11:19:58.064 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:19:58.064 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f746802ff60 msgr2=0x7f7468032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f746802ff60 0x7f7468032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f74740096a0 tx=0x7f7474006b40).stop 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 msgr2=0x7f7480106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f7480106690 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f7470013510 tx=0x7f747000b040).stop 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 shutdown_connections 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f746802ff60 0x7f7468032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:58.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 --2- 172.21.15.178:0/291954111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7480060280 0x7f7480106690 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:19:58.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 >> 172.21.15.178:0/291954111 conn(0x7f74800f0b70 msgr2=0x7f74800f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:19:58.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 shutdown_connections 2024-01-21T11:19:58.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:19:58.065+0000 7f74863c9700 1 -- 172.21.15.178:0/291954111 wait complete. 2024-01-21T11:19:58.067 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:19:59.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:57.701105+0000 mgr.smithi089.ufybqw (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:19:59.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:19:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:19:58.065276+0000 mon.smithi089 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.178:0/291954111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:19:59.464 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:19:59.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 -- 172.21.15.178:0/1727035860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 msgr2=0x7fb3e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 --2- 172.21.15.178:0/1727035860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e80f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fb3d8009fb0 tx=0x7fb3d8009b30).stop 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 -- 172.21.15.178:0/1727035860 shutdown_connections 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 --2- 172.21.15.178:0/1727035860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e80f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 -- 172.21.15.178:0/1727035860 >> 172.21.15.178:0/1727035860 conn(0x7fb3e80f0b70 msgr2=0x7fb3e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 -- 172.21.15.178:0/1727035860 shutdown_connections 2024-01-21T11:20:00.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 -- 172.21.15.178:0/1727035860 wait complete. 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.614+0000 7fb3efa50700 1 Processor -- start 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3efa50700 1 -- start start 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3efa50700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3efa50700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8060220 con 0x7fb3e80f5410 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3ed7ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3ed7ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:46032/0 (socket says 172.21.15.178:46032) 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3ed7ec700 1 -- 172.21.15.178:0/1893271140 learned_addr learned my addr 172.21.15.178:0/1893271140 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3ed7ec700 1 -- 172.21.15.178:0/1893271140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8005040 con 0x7fb3e80f5410 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3ed7ec700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e8109450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb3d8013510 tx=0x7fb3d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.615+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d8009320 con 0x7fb3e80f5410 2024-01-21T11:20:00.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.616+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e8060420 con 0x7fb3e80f5410 2024-01-21T11:20:00.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.616+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3d8013f00 con 0x7fb3e80f5410 2024-01-21T11:20:00.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.616+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d8014d90 con 0x7fb3e80f5410 2024-01-21T11:20:00.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.616+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8109b00 con 0x7fb3e80f5410 2024-01-21T11:20:00.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.617+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3d80095f0 con 0x7fb3e80f5410 2024-01-21T11:20:00.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.617+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e8059100 con 0x7fb3e80f5410 2024-01-21T11:20:00.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.617+0000 7fb3ddffb700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3d402ff60 0x7fb3d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:00.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.617+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3d8041000 con 0x7fb3e80f5410 2024-01-21T11:20:00.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.617+0000 7fb3ecfeb700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3d402ff60 0x7fb3d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:00.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.618+0000 7fb3ecfeb700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3d402ff60 0x7fb3d4032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb3e40096a0 tx=0x7fb3e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:00.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.622+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3d8014420 con 0x7fb3e80f5410 2024-01-21T11:20:00.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.831+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3e8045ad0 con 0x7fb3e80f5410 2024-01-21T11:20:00.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.832+0000 7fb3ddffb700 1 -- 172.21.15.178:0/1893271140 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3d801d7f0 con 0x7fb3e80f5410 2024-01-21T11:20:00.833 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:00.833 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3d402ff60 msgr2=0x7fb3d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3d402ff60 0x7fb3d4032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb3e40096a0 tx=0x7fb3e4006b40).stop 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 msgr2=0x7fb3e8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e8109450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb3d8013510 tx=0x7fb3d800b040).stop 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 shutdown_connections 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb3d402ff60 0x7fb3d4032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 --2- 172.21.15.178:0/1893271140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e80f5410 0x7fb3e8109450 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 >> 172.21.15.178:0/1893271140 conn(0x7fb3e80f0b70 msgr2=0x7fb3e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:00.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 shutdown_connections 2024-01-21T11:20:00.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:00.834+0000 7fb3efa50700 1 -- 172.21.15.178:0/1893271140 wait complete. 2024-01-21T11:20:00.836 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:01.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:19:59.701352+0000 mgr.smithi089.ufybqw (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:02.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:00.833947+0000 mon.smithi089 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.178:0/1893271140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:02.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:02.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:01.701653+0000 mgr.smithi089.ufybqw (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:03.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 -- 172.21.15.178:0/4209443648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 msgr2=0x7f85040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 --2- 172.21.15.178:0/4209443648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85040f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f84f8009fb0 tx=0x7f84f8009b30).stop 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 -- 172.21.15.178:0/4209443648 shutdown_connections 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 --2- 172.21.15.178:0/4209443648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85040f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 -- 172.21.15.178:0/4209443648 >> 172.21.15.178:0/4209443648 conn(0x7f85040f09a0 msgr2=0x7f85040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 -- 172.21.15.178:0/4209443648 shutdown_connections 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 -- 172.21.15.178:0/4209443648 wait complete. 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 Processor -- start 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.412+0000 7f8509b8d700 1 -- start start 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.413+0000 7f8509b8d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85041064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.413+0000 7f8509b8d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504106990 con 0x7f85040f51a0 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.413+0000 7f8508b8b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85041064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.413+0000 7f8508b8b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85041064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:38876/0 (socket says 172.21.15.178:38876) 2024-01-21T11:20:03.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.413+0000 7f8508b8b700 1 -- 172.21.15.178:0/2134506058 learned_addr learned my addr 172.21.15.178:0/2134506058 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.413+0000 7f8508b8b700 1 -- 172.21.15.178:0/2134506058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8005040 con 0x7f85040f51a0 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.414+0000 7f8508b8b700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85041064a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f84f8013510 tx=0x7f84f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.414+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f8009320 con 0x7f85040f51a0 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.414+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8504106b90 con 0x7f85040f51a0 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.414+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84f8013f00 con 0x7f85040f51a0 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.414+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f8014dc0 con 0x7f85040f51a0 2024-01-21T11:20:03.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.414+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8504107020 con 0x7f85040f51a0 2024-01-21T11:20:03.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.415+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84f80095f0 con 0x7f85040f51a0 2024-01-21T11:20:03.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.415+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8504045ad0 con 0x7f85040f51a0 2024-01-21T11:20:03.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.415+0000 7f85017fa700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f84f402ff60 0x7f84f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:03.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.415+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84f8041040 con 0x7f85040f51a0 2024-01-21T11:20:03.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.415+0000 7f8503fff700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f84f402ff60 0x7f84f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:03.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.416+0000 7f8503fff700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f84f402ff60 0x7f84f4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f84f00096a0 tx=0x7f84f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:03.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.420+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84f8014420 con 0x7f85040f51a0 2024-01-21T11:20:03.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.624+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8504107ed0 con 0x7f85040f51a0 2024-01-21T11:20:03.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.626+0000 7f85017fa700 1 -- 172.21.15.178:0/2134506058 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84f8010230 con 0x7f85040f51a0 2024-01-21T11:20:03.626 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:03.626 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:03.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f84f402ff60 msgr2=0x7f84f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f84f402ff60 0x7f84f4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f84f00096a0 tx=0x7f84f0006b40).stop 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 msgr2=0x7f85041064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85041064a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f84f8013510 tx=0x7f84f800b040).stop 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 shutdown_connections 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f84f402ff60 0x7f84f4032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 --2- 172.21.15.178:0/2134506058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85040f51a0 0x7f85041064a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 >> 172.21.15.178:0/2134506058 conn(0x7f85040f09a0 msgr2=0x7f850405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 shutdown_connections 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:03.627+0000 7f8509b8d700 1 -- 172.21.15.178:0/2134506058 wait complete. 2024-01-21T11:20:03.628 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:04.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:03 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:03.626731+0000 mon.smithi089 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.178:0/2134506058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:05.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:05.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:05.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:03.701982+0000 mgr.smithi089.ufybqw (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 -- 172.21.15.178:0/1316853761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f4740 msgr2=0x7fce400f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 --2- 172.21.15.178:0/1316853761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f4740 0x7fce400f4b60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fce30009fb0 tx=0x7fce30009b30).stop 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 -- 172.21.15.178:0/1316853761 shutdown_connections 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 --2- 172.21.15.178:0/1316853761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f4740 0x7fce400f4b60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 -- 172.21.15.178:0/1316853761 >> 172.21.15.178:0/1316853761 conn(0x7fce400f0260 msgr2=0x7fce400f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 -- 172.21.15.178:0/1316853761 shutdown_connections 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.185+0000 7fce47b42700 1 -- 172.21.15.178:0/1316853761 wait complete. 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce47b42700 1 Processor -- start 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce47b42700 1 -- start start 2024-01-21T11:20:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce47b42700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 0x7fce400f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce47b42700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400f76d0 con 0x7fce400f72b0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce458de700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 0x7fce400f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce458de700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 0x7fce400f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:38882/0 (socket says 172.21.15.178:38882) 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce458de700 1 -- 172.21.15.178:0/3454315418 learned_addr learned my addr 172.21.15.178:0/3454315418 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.186+0000 7fce458de700 1 -- 172.21.15.178:0/3454315418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce30005040 con 0x7fce400f72b0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.187+0000 7fce458de700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 0x7fce400f5770 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fce30009f80 tx=0x7fce30009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.187+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce300136c0 con 0x7fce400f72b0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.187+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce400f5c60 con 0x7fce400f72b0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.187+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce30013e70 con 0x7fce400f72b0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.187+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce30014420 con 0x7fce400f72b0 2024-01-21T11:20:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.187+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce400f6150 con 0x7fce400f72b0 2024-01-21T11:20:06.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.188+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce30025060 con 0x7fce400f72b0 2024-01-21T11:20:06.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.188+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce40045ad0 con 0x7fce400f72b0 2024-01-21T11:20:06.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.188+0000 7fce367fc700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fce2c02ff60 0x7fce2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:06.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.188+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce30040e00 con 0x7fce400f72b0 2024-01-21T11:20:06.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.188+0000 7fce450dd700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fce2c02ff60 0x7fce2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:06.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.188+0000 7fce450dd700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fce2c02ff60 0x7fce2c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fce3c0096a0 tx=0x7fce3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:06.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.192+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce3001d3a0 con 0x7fce400f72b0 2024-01-21T11:20:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.400+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce4010c9f0 con 0x7fce400f72b0 2024-01-21T11:20:06.400 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.401+0000 7fce367fc700 1 -- 172.21.15.178:0/3454315418 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce3002d2d0 con 0x7fce400f72b0 2024-01-21T11:20:06.401 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:06.401 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.402+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fce2c02ff60 msgr2=0x7fce2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.402+0000 7fce47b42700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fce2c02ff60 0x7fce2c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fce3c0096a0 tx=0x7fce3c006b40).stop 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.402+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 msgr2=0x7fce400f5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.402+0000 7fce47b42700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 0x7fce400f5770 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fce30009f80 tx=0x7fce30009b30).stop 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.403+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 shutdown_connections 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.403+0000 7fce47b42700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fce2c02ff60 0x7fce2c032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.403+0000 7fce47b42700 1 --2- 172.21.15.178:0/3454315418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce400f72b0 0x7fce400f5770 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.403+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 >> 172.21.15.178:0/3454315418 conn(0x7fce400f0260 msgr2=0x7fce400f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.403+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 shutdown_connections 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:06.403+0000 7fce47b42700 1 -- 172.21.15.178:0/3454315418 wait complete. 2024-01-21T11:20:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:07.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:05.702363+0000 mgr.smithi089.ufybqw (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:07.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:06.401572+0000 mon.smithi089 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.178:0/3454315418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:07.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:07.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 -- 172.21.15.178:0/208914071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 msgr2=0x7f761c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 --2- 172.21.15.178:0/208914071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c0f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f7610009fb0 tx=0x7f7610009b30).stop 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 -- 172.21.15.178:0/208914071 shutdown_connections 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 --2- 172.21.15.178:0/208914071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c0f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 -- 172.21.15.178:0/208914071 >> 172.21.15.178:0/208914071 conn(0x7f761c0f0b70 msgr2=0x7f761c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 -- 172.21.15.178:0/208914071 shutdown_connections 2024-01-21T11:20:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.960+0000 7f7620ae6700 1 -- 172.21.15.178:0/208914071 wait complete. 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f7620ae6700 1 Processor -- start 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f7620ae6700 1 -- start start 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f7620ae6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f7620ae6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c060220 con 0x7f761c0f5410 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f761a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f761a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:38892/0 (socket says 172.21.15.178:38892) 2024-01-21T11:20:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.961+0000 7f761a59c700 1 -- 172.21.15.178:0/227980870 learned_addr learned my addr 172.21.15.178:0/227980870 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:08.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f761a59c700 1 -- 172.21.15.178:0/227980870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610005040 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f761a59c700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c109420 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f7610014000 tx=0x7f761000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76100095f0 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c060420 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7610007e70 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7610014ea0 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.962+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c109ad0 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.963+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c059100 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.963+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7610025070 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.963+0000 7f760affd700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f760402ff60 0x7f7604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.963+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7610040970 con 0x7f761c0f5410 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.963+0000 7f7619d9b700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f760402ff60 0x7f7604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:08.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.964+0000 7f7619d9b700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f760402ff60 0x7f7604032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f760c0096a0 tx=0x7f760c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:08.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:08.968+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76100079c0 con 0x7f761c0f5410 2024-01-21T11:20:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.173+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f761c045ad0 con 0x7f761c0f5410 2024-01-21T11:20:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.174+0000 7f760affd700 1 -- 172.21.15.178:0/227980870 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f761000f300 con 0x7f761c0f5410 2024-01-21T11:20:09.174 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:09.174 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:09.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f760402ff60 msgr2=0x7f7604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:09.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f760402ff60 0x7f7604032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f760c0096a0 tx=0x7f760c006b40).stop 2024-01-21T11:20:09.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 msgr2=0x7f761c109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:09.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c109420 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f7610014000 tx=0x7f761000b040).stop 2024-01-21T11:20:09.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 shutdown_connections 2024-01-21T11:20:09.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f760402ff60 0x7f7604032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:09.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 --2- 172.21.15.178:0/227980870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f761c0f5410 0x7f761c109420 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:09.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 >> 172.21.15.178:0/227980870 conn(0x7f761c0f0b70 msgr2=0x7f761c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:09.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.175+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 shutdown_connections 2024-01-21T11:20:09.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:09.176+0000 7f7620ae6700 1 -- 172.21.15.178:0/227980870 wait complete. 2024-01-21T11:20:09.176 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:09.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:07.702738+0000 mgr.smithi089.ufybqw (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:10.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:09 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:09.174723+0000 mon.smithi089 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.178:0/227980870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:10.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:10.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:11.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:09.703001+0000 mgr.smithi089.ufybqw (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:11.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.695+0000 7f63cf4a4700 1 -- 172.21.15.178:0/3377309713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 msgr2=0x7f63c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:11.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.695+0000 7f63cf4a4700 1 --2- 172.21.15.178:0/3377309713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c80f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f63b8009fb0 tx=0x7f63b8009b30).stop 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.695+0000 7f63cf4a4700 1 -- 172.21.15.178:0/3377309713 shutdown_connections 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.695+0000 7f63cf4a4700 1 --2- 172.21.15.178:0/3377309713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c80f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.695+0000 7f63cf4a4700 1 -- 172.21.15.178:0/3377309713 >> 172.21.15.178:0/3377309713 conn(0x7f63c80f0b70 msgr2=0x7f63c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cf4a4700 1 -- 172.21.15.178:0/3377309713 shutdown_connections 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cf4a4700 1 -- 172.21.15.178:0/3377309713 wait complete. 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cf4a4700 1 Processor -- start 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cf4a4700 1 -- start start 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cf4a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cf4a4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8060210 con 0x7f63c80f5410 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.696+0000 7f63cd240700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:11.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63cd240700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:38912/0 (socket says 172.21.15.178:38912) 2024-01-21T11:20:11.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63cd240700 1 -- 172.21.15.178:0/1479159963 learned_addr learned my addr 172.21.15.178:0/1479159963 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:11.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63cd240700 1 -- 172.21.15.178:0/1479159963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8005040 con 0x7f63c80f5410 2024-01-21T11:20:11.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63cd240700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c81093f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f63b8013510 tx=0x7f63b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8009320 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c8060410 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b8013f00 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8014d90 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.697+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c8109aa0 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.698+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63b80095f0 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.698+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c8059100 con 0x7f63c80f5410 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.698+0000 7f63bdffb700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63b402ffb0 0x7f63b4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:11.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.698+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63b8040f80 con 0x7f63c80f5410 2024-01-21T11:20:11.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.698+0000 7f63cca3f700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63b402ffb0 0x7f63b4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:11.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.699+0000 7f63cca3f700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63b402ffb0 0x7f63b4032470 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f63c40096a0 tx=0x7f63c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:11.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.703+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63b8014420 con 0x7f63c80f5410 2024-01-21T11:20:11.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.910+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63c8045ad0 con 0x7f63c80f5410 2024-01-21T11:20:11.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.911+0000 7f63bdffb700 1 -- 172.21.15.178:0/1479159963 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63b801d7f0 con 0x7f63c80f5410 2024-01-21T11:20:11.911 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:11.911 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:11.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63b402ffb0 msgr2=0x7f63b4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:11.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63b402ffb0 0x7f63b4032470 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f63c40096a0 tx=0x7f63c4006b40).stop 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 msgr2=0x7f63c81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c81093f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f63b8013510 tx=0x7f63b800b040).stop 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 shutdown_connections 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f63b402ffb0 0x7f63b4032470 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 --2- 172.21.15.178:0/1479159963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c80f5410 0x7f63c81093f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 >> 172.21.15.178:0/1479159963 conn(0x7f63c80f0b70 msgr2=0x7f63c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 shutdown_connections 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:11.913+0000 7f63cf4a4700 1 -- 172.21.15.178:0/1479159963 wait complete. 2024-01-21T11:20:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:12.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:11 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:11.703232+0000 mgr.smithi089.ufybqw (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:13.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:11.911730+0000 mon.smithi089 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.178:0/1479159963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:13.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:13.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:14.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:13.703543+0000 mgr.smithi089.ufybqw (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:14.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3341649231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 msgr2=0x7fd1bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 --2- 172.21.15.178:0/3341649231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc0f66c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fd1ac009fb0 tx=0x7fd1ac009b30).stop 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3341649231 shutdown_connections 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 --2- 172.21.15.178:0/3341649231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc0f66c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3341649231 >> 172.21.15.178:0/3341649231 conn(0x7fd1bc0f1aa0 msgr2=0x7fd1bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3341649231 shutdown_connections 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3341649231 wait complete. 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 Processor -- start 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.435+0000 7fd1c16d3700 1 -- start start 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1c16d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1c16d3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1069a0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1bbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:14.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1bbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:60512/0 (socket says 172.21.15.178:60512) 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1bbfff700 1 -- 172.21.15.178:0/3405316472 learned_addr learned my addr 172.21.15.178:0/3405316472 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1bbfff700 1 -- 172.21.15.178:0/3405316472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac005040 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1bbfff700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc1064b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fd1ac013510 tx=0x7fd1ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1ac009320 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc106ba0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1ac013f00 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1ac014dc0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.436+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc107030 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.437+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1ac0095f0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.437+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc045ad0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.437+0000 7fd1b8ff9700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd1a402ff60 0x7fd1a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:14.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.437+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1ac0410c0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.438+0000 7fd1bb7fe700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd1a402ff60 0x7fd1a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:14.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.438+0000 7fd1bb7fe700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd1a402ff60 0x7fd1a4032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd1b00096a0 tx=0x7fd1b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:14.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.442+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1ac01c790 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.648+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1bc107ee0 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.649+0000 7fd1b8ff9700 1 -- 172.21.15.178:0/3405316472 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1ac010230 con 0x7fd1bc0f62a0 2024-01-21T11:20:14.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:14.649 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:14.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd1a402ff60 msgr2=0x7fd1a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd1a402ff60 0x7fd1a4032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd1b00096a0 tx=0x7fd1b0006b40).stop 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 msgr2=0x7fd1bc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc1064b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fd1ac013510 tx=0x7fd1ac00b040).stop 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 shutdown_connections 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd1a402ff60 0x7fd1a4032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 --2- 172.21.15.178:0/3405316472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1bc0f62a0 0x7fd1bc1064b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.650+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 >> 172.21.15.178:0/3405316472 conn(0x7fd1bc0f1aa0 msgr2=0x7fd1bc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.651+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 shutdown_connections 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:14.651+0000 7fd1c16d3700 1 -- 172.21.15.178:0/3405316472 wait complete. 2024-01-21T11:20:14.651 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:13.777570+0000 mon.smithi089 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:20:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:13.786346+0000 mon.smithi089 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:20:14.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:14 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:14.649569+0000 mon.smithi089 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.178:0/3405316472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:16.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:16.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:16.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:15.703893+0000 mgr.smithi089.ufybqw (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:17.263 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 -- 172.21.15.178:0/504272008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 msgr2=0x7f0bb40f8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 --2- 172.21.15.178:0/504272008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb40f8f50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f0b9c009fb0 tx=0x7f0b9c009b30).stop 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 -- 172.21.15.178:0/504272008 shutdown_connections 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 --2- 172.21.15.178:0/504272008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb40f8f50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 -- 172.21.15.178:0/504272008 >> 172.21.15.178:0/504272008 conn(0x7f0bb40f0930 msgr2=0x7f0bb40f2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 -- 172.21.15.178:0/504272008 shutdown_connections 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.262+0000 7f0bb977e700 1 -- 172.21.15.178:0/504272008 wait complete. 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb977e700 1 Processor -- start 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb977e700 1 -- start start 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb977e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb4106550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb977e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4106a40 con 0x7f0bb40601e0 2024-01-21T11:20:17.264 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb4106550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb4106550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:60524/0 (socket says 172.21.15.178:60524) 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.263+0000 7f0bb2ffd700 1 -- 172.21.15.178:0/1647373008 learned_addr learned my addr 172.21.15.178:0/1647373008 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bb2ffd700 1 -- 172.21.15.178:0/1647373008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c005040 con 0x7f0bb40601e0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bb2ffd700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb4106550 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f0b9c004dd0 tx=0x7f0b9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c018070 con 0x7f0bb40601e0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b9c013410 con 0x7f0bb40601e0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb4106c40 con 0x7f0bb40601e0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c01c3c0 con 0x7f0bb40601e0 2024-01-21T11:20:17.265 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.264+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb41070d0 con 0x7f0bb40601e0 2024-01-21T11:20:17.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.265+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b9c01d420 con 0x7f0bb40601e0 2024-01-21T11:20:17.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.265+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb4045ad0 con 0x7f0bb40601e0 2024-01-21T11:20:17.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.265+0000 7f0bab7fe700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0ba002ff60 0x7f0ba0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:17.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.265+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b9c041330 con 0x7f0bb40601e0 2024-01-21T11:20:17.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.265+0000 7f0bb27fc700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0ba002ff60 0x7f0ba0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:17.266 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.266+0000 7f0bb27fc700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0ba002ff60 0x7f0ba0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0ba40096a0 tx=0x7f0ba4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:17.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.269+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b9c01d7e0 con 0x7f0bb40601e0 2024-01-21T11:20:17.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.476+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bb4107ed0 con 0x7f0bb40601e0 2024-01-21T11:20:17.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.476+0000 7f0bab7fe700 1 -- 172.21.15.178:0/1647373008 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b9c01b370 con 0x7f0bb40601e0 2024-01-21T11:20:17.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:17.476 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0ba002ff60 msgr2=0x7f0ba0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0ba002ff60 0x7f0ba0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0ba40096a0 tx=0x7f0ba4006b40).stop 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 msgr2=0x7f0bb4106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb4106550 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f0b9c004dd0 tx=0x7f0b9c009b30).stop 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 shutdown_connections 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f0ba002ff60 0x7f0ba0032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 --2- 172.21.15.178:0/1647373008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40601e0 0x7f0bb4106550 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 >> 172.21.15.178:0/1647373008 conn(0x7f0bb40f0930 msgr2=0x7f0bb40f14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 shutdown_connections 2024-01-21T11:20:17.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:17.478+0000 7f0bb977e700 1 -- 172.21.15.178:0/1647373008 wait complete. 2024-01-21T11:20:17.479 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:17.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:17 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:17.477043+0000 mon.smithi089 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.178:0/1647373008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:18.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:18.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:18.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:17.704239+0000 mgr.smithi089.ufybqw (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 -- 172.21.15.178:0/152015527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 msgr2=0x7f5cfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 --2- 172.21.15.178:0/152015527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5cec009fb0 tx=0x7f5cec009b30).stop 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 -- 172.21.15.178:0/152015527 shutdown_connections 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 --2- 172.21.15.178:0/152015527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 -- 172.21.15.178:0/152015527 >> 172.21.15.178:0/152015527 conn(0x7f5cfc0f0b70 msgr2=0x7f5cfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 -- 172.21.15.178:0/152015527 shutdown_connections 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.020+0000 7f5d02486700 1 -- 172.21.15.178:0/152015527 wait complete. 2024-01-21T11:20:20.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5d02486700 1 Processor -- start 2024-01-21T11:20:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5d02486700 1 -- start start 2024-01-21T11:20:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5d02486700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:20.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5d02486700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc105af0 con 0x7f5cfc0f5410 2024-01-21T11:20:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5cfbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5cfbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:60544/0 (socket says 172.21.15.178:60544) 2024-01-21T11:20:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.021+0000 7f5cfbfff700 1 -- 172.21.15.178:0/1077773624 learned_addr learned my addr 172.21.15.178:0/1077773624 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:20.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5cfbfff700 1 -- 172.21.15.178:0/1077773624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec005040 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5cfbfff700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc105600 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5cec013510 tx=0x7f5cec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cec009320 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cfc105cf0 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cec013f00 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cec014d90 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.022+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc106180 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.023+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cec0095f0 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.023+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc045ad0 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.023+0000 7f5cf8ff9700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5ce802ff60 0x7f5ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.023+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cec0410c0 con 0x7f5cfc0f5410 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.023+0000 7f5cfb7fe700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5ce802ff60 0x7f5ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:20.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.024+0000 7f5cfb7fe700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5ce802ff60 0x7f5ce8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5ce40096a0 tx=0x7f5ce4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:20.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.028+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cec01d7f0 con 0x7f5cfc0f5410 2024-01-21T11:20:20.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.233+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cfc0fae60 con 0x7f5cfc0f5410 2024-01-21T11:20:20.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.234+0000 7f5cf8ff9700 1 -- 172.21.15.178:0/1077773624 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cec010230 con 0x7f5cfc0f5410 2024-01-21T11:20:20.234 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:20.234 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:20.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5ce802ff60 msgr2=0x7f5ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:20.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5ce802ff60 0x7f5ce8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5ce40096a0 tx=0x7f5ce4006b40).stop 2024-01-21T11:20:20.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 msgr2=0x7f5cfc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:20.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc105600 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5cec013510 tx=0x7f5cec00b040).stop 2024-01-21T11:20:20.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 shutdown_connections 2024-01-21T11:20:20.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5ce802ff60 0x7f5ce8032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:20.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 --2- 172.21.15.178:0/1077773624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cfc0f5410 0x7f5cfc105600 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:20.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 >> 172.21.15.178:0/1077773624 conn(0x7f5cfc0f0b70 msgr2=0x7f5cfc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:20.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 shutdown_connections 2024-01-21T11:20:20.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:20.235+0000 7f5d02486700 1 -- 172.21.15.178:0/1077773624 wait complete. 2024-01-21T11:20:20.236 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:21.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:19.704514+0000 mgr.smithi089.ufybqw (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:21.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:20.234477+0000 mon.smithi089 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.178:0/1077773624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:21.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:21.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:22.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.756+0000 7f11c580d700 1 -- 172.21.15.178:0/2649984278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 msgr2=0x7f11c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:22.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.756+0000 7f11c580d700 1 --2- 172.21.15.178:0/2649984278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c00f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f11b4009fb0 tx=0x7f11b4009b30).stop 2024-01-21T11:20:22.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.756+0000 7f11c580d700 1 -- 172.21.15.178:0/2649984278 shutdown_connections 2024-01-21T11:20:22.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.756+0000 7f11c580d700 1 --2- 172.21.15.178:0/2649984278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c00f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:22.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.756+0000 7f11c580d700 1 -- 172.21.15.178:0/2649984278 >> 172.21.15.178:0/2649984278 conn(0x7f11c00f1c70 msgr2=0x7f11c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:22.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.757+0000 7f11c580d700 1 -- 172.21.15.178:0/2649984278 shutdown_connections 2024-01-21T11:20:22.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.757+0000 7f11c580d700 1 -- 172.21.15.178:0/2649984278 wait complete. 2024-01-21T11:20:22.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.757+0000 7f11c580d700 1 Processor -- start 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.757+0000 7f11c580d700 1 -- start start 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.757+0000 7f11c580d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.757+0000 7f11c580d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0106c20 con 0x7f11c00f6510 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55926/0 (socket says 172.21.15.178:55926) 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11beffd700 1 -- 172.21.15.178:0/2958030153 learned_addr learned my addr 172.21.15.178:0/2958030153 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11beffd700 1 -- 172.21.15.178:0/2958030153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b4005040 con 0x7f11c00f6510 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11beffd700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f11b4013510 tx=0x7f11b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11b4009320 con 0x7f11c00f6510 2024-01-21T11:20:22.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c0106e20 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11b4013f00 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11b4014d90 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.758+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c01072b0 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.759+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11b40095f0 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.759+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c0045ad0 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.759+0000 7f11a77fe700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f11a802ff60 0x7f11a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.759+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11b40410a0 con 0x7f11c00f6510 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.759+0000 7f11be7fc700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f11a802ff60 0x7f11a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:22.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.760+0000 7f11be7fc700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f11a802ff60 0x7f11a8032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f11b00096a0 tx=0x7f11b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:22.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.764+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11b401d7f0 con 0x7f11c00f6510 2024-01-21T11:20:22.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.974+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11c00fbf60 con 0x7f11c00f6510 2024-01-21T11:20:22.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.975+0000 7f11a77fe700 1 -- 172.21.15.178:0/2958030153 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11b4010230 con 0x7f11c00f6510 2024-01-21T11:20:22.974 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:22.975 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:22.976 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f11a802ff60 msgr2=0x7f11a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:22.976 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f11a802ff60 0x7f11a8032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f11b00096a0 tx=0x7f11b0006b40).stop 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 msgr2=0x7f11c0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f11b4013510 tx=0x7f11b400b040).stop 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 shutdown_connections 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f11a802ff60 0x7f11a8032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 --2- 172.21.15.178:0/2958030153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.976+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 >> 172.21.15.178:0/2958030153 conn(0x7f11c00f1c70 msgr2=0x7f11c00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.977+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 shutdown_connections 2024-01-21T11:20:22.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:22.977+0000 7f11c580d700 1 -- 172.21.15.178:0/2958030153 wait complete. 2024-01-21T11:20:22.978 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:23.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:21.704856+0000 mgr.smithi089.ufybqw (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:24.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:23 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:22.975470+0000 mon.smithi089 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.178:0/2958030153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:24.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:24.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:25.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:23.705149+0000 mgr.smithi089.ufybqw (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:25.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 -- 172.21.15.178:0/3945337181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 msgr2=0x7faccc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:25.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 --2- 172.21.15.178:0/3945337181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc0f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7facb4009fb0 tx=0x7facb4009b30).stop 2024-01-21T11:20:25.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 -- 172.21.15.178:0/3945337181 shutdown_connections 2024-01-21T11:20:25.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 --2- 172.21.15.178:0/3945337181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc0f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:25.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 -- 172.21.15.178:0/3945337181 >> 172.21.15.178:0/3945337181 conn(0x7faccc0f1c70 msgr2=0x7faccc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:25.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 -- 172.21.15.178:0/3945337181 shutdown_connections 2024-01-21T11:20:25.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.566+0000 7facd152f700 1 -- 172.21.15.178:0/3945337181 wait complete. 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7facd152f700 1 Processor -- start 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7facd152f700 1 -- start start 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7facd152f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7facd152f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc106c20 con 0x7faccc0f6510 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7faccaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7faccaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55938/0 (socket says 172.21.15.178:55938) 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.567+0000 7faccaffd700 1 -- 172.21.15.178:0/1086391760 learned_addr learned my addr 172.21.15.178:0/1086391760 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7faccaffd700 1 -- 172.21.15.178:0/1086391760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb4005040 con 0x7faccc0f6510 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7faccaffd700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7facb4013510 tx=0x7facb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facb4009320 con 0x7faccc0f6510 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facb4013f00 con 0x7faccc0f6510 2024-01-21T11:20:25.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc106e20 con 0x7faccc0f6510 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facb4014d90 con 0x7faccc0f6510 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.568+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc1072b0 con 0x7faccc0f6510 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.569+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facb40095f0 con 0x7faccc0f6510 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.569+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc045ad0 con 0x7faccc0f6510 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.569+0000 7facc37fe700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facb802ffb0 0x7facb8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.569+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facb4040fe0 con 0x7faccc0f6510 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.569+0000 7facca7fc700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facb802ffb0 0x7facb8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:25.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.570+0000 7facca7fc700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facb802ffb0 0x7facb8032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7facbc0096a0 tx=0x7facbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:25.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.574+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facb406d2b0 con 0x7faccc0f6510 2024-01-21T11:20:25.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.780+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faccc0fbf60 con 0x7faccc0f6510 2024-01-21T11:20:25.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.780+0000 7facc37fe700 1 -- 172.21.15.178:0/1086391760 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facb4014420 con 0x7faccc0f6510 2024-01-21T11:20:25.780 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:25.780 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facb802ffb0 msgr2=0x7facb8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facb802ffb0 0x7facb8032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7facbc0096a0 tx=0x7facbc006b40).stop 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 msgr2=0x7faccc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7facb4013510 tx=0x7facb400b040).stop 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 shutdown_connections 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7facb802ffb0 0x7facb8032470 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 --2- 172.21.15.178:0/1086391760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc0f6510 0x7faccc106730 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 >> 172.21.15.178:0/1086391760 conn(0x7faccc0f1c70 msgr2=0x7faccc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 shutdown_connections 2024-01-21T11:20:25.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:25.782+0000 7facd152f700 1 -- 172.21.15.178:0/1086391760 wait complete. 2024-01-21T11:20:25.783 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:27.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:27.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:27.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:25.705464+0000 mgr.smithi089.ufybqw (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:27.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:25.781227+0000 mon.smithi089 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.178:0/1086391760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:28.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.322+0000 7f63a87bf700 1 -- 172.21.15.178:0/539869658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 msgr2=0x7f63a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:28.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.322+0000 7f63a87bf700 1 --2- 172.21.15.178:0/539869658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a00f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f6390009fb0 tx=0x7f6390009b30).stop 2024-01-21T11:20:28.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 -- 172.21.15.178:0/539869658 shutdown_connections 2024-01-21T11:20:28.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 --2- 172.21.15.178:0/539869658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a00f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:28.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 -- 172.21.15.178:0/539869658 >> 172.21.15.178:0/539869658 conn(0x7f63a00f0b70 msgr2=0x7f63a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 -- 172.21.15.178:0/539869658 shutdown_connections 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 -- 172.21.15.178:0/539869658 wait complete. 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 Processor -- start 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 -- start start 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.323+0000 7f63a87bf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0060240 con 0x7f63a00f5410 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a655b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a655b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55956/0 (socket says 172.21.15.178:55956) 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a655b700 1 -- 172.21.15.178:0/163132522 learned_addr learned my addr 172.21.15.178:0/163132522 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a655b700 1 -- 172.21.15.178:0/163132522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6390005040 con 0x7f63a00f5410 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a655b700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6390013510 tx=0x7f639000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6390009320 con 0x7f63a00f5410 2024-01-21T11:20:28.326 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a0060440 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6390013f00 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6390014d90 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.324+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a0109ae0 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.325+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63900095f0 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.325+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a0045ad0 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.325+0000 7f6396ffd700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f638c02ff60 0x7f638c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.325+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63900410a0 con 0x7f63a00f5410 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.325+0000 7f63a5d5a700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f638c02ff60 0x7f638c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:28.327 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.326+0000 7f63a5d5a700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f638c02ff60 0x7f638c032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f639c0096a0 tx=0x7f639c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:28.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.330+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f639001d7f0 con 0x7f63a00f5410 2024-01-21T11:20:28.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.535+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63a00fae60 con 0x7f63a00f5410 2024-01-21T11:20:28.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.535+0000 7f6396ffd700 1 -- 172.21.15.178:0/163132522 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6390010230 con 0x7f63a00f5410 2024-01-21T11:20:28.535 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:28.536 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f638c02ff60 msgr2=0x7f638c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f638c02ff60 0x7f638c032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f639c0096a0 tx=0x7f639c006b40).stop 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 msgr2=0x7f63a0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6390013510 tx=0x7f639000b040).stop 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 shutdown_connections 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f638c02ff60 0x7f638c032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 --2- 172.21.15.178:0/163132522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 >> 172.21.15.178:0/163132522 conn(0x7f63a00f0b70 msgr2=0x7f63a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 shutdown_connections 2024-01-21T11:20:28.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:28.537+0000 7f63a87bf700 1 -- 172.21.15.178:0/163132522 wait complete. 2024-01-21T11:20:28.538 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:29.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:27.705785+0000 mgr.smithi089.ufybqw (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:29.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:28.536259+0000 mon.smithi089 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.178:0/163132522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:29.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:29.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:31.030 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 -- 172.21.15.178:0/2736738027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 msgr2=0x7f48b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 --2- 172.21.15.178:0/2736738027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b40f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f48a8009fb0 tx=0x7f48a8009b30).stop 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 -- 172.21.15.178:0/2736738027 shutdown_connections 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 --2- 172.21.15.178:0/2736738027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b40f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 -- 172.21.15.178:0/2736738027 >> 172.21.15.178:0/2736738027 conn(0x7f48b40f0b70 msgr2=0x7f48b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 -- 172.21.15.178:0/2736738027 shutdown_connections 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 -- 172.21.15.178:0/2736738027 wait complete. 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 Processor -- start 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.029+0000 7f48baf90700 1 -- start start 2024-01-21T11:20:31.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48baf90700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48baf90700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b4105b20 con 0x7f48b40f5410 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48b8d2c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48b8d2c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55976/0 (socket says 172.21.15.178:55976) 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48b8d2c700 1 -- 172.21.15.178:0/3365791647 learned_addr learned my addr 172.21.15.178:0/3365791647 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48b8d2c700 1 -- 172.21.15.178:0/3365791647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a8005040 con 0x7f48b40f5410 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.030+0000 7f48b8d2c700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b4105630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f48a8013510 tx=0x7f48a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.031+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48a8009320 con 0x7f48b40f5410 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.031+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b4105d20 con 0x7f48b40f5410 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.031+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48a8013f00 con 0x7f48b40f5410 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.031+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48a8014d90 con 0x7f48b40f5410 2024-01-21T11:20:31.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.031+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b41061b0 con 0x7f48b40f5410 2024-01-21T11:20:31.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.031+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48a80095f0 con 0x7f48b40f5410 2024-01-21T11:20:31.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.032+0000 7f48b17fa700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f48a402ff60 0x7f48a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:31.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.032+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b4045ad0 con 0x7f48b40f5410 2024-01-21T11:20:31.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.032+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48a8041030 con 0x7f48b40f5410 2024-01-21T11:20:31.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.032+0000 7f48b3fff700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f48a402ff60 0x7f48a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:31.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.032+0000 7f48b3fff700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f48a402ff60 0x7f48a4032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:31.036 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.036+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48a806b2b0 con 0x7f48b40f5410 2024-01-21T11:20:31.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:29.706060+0000 mgr.smithi089.ufybqw (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:31.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.247+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48b40faf50 con 0x7f48b40f5410 2024-01-21T11:20:31.247 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.247+0000 7f48b17fa700 1 -- 172.21.15.178:0/3365791647 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f48a802c090 con 0x7f48b40f5410 2024-01-21T11:20:31.247 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:31.247 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f48a402ff60 msgr2=0x7f48a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f48a402ff60 0x7f48a4032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).stop 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 msgr2=0x7f48b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b4105630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f48a8013510 tx=0x7f48a800b040).stop 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 shutdown_connections 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f48a402ff60 0x7f48a4032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 --2- 172.21.15.178:0/3365791647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48b40f5410 0x7f48b4105630 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 >> 172.21.15.178:0/3365791647 conn(0x7f48b40f0b70 msgr2=0x7f48b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:31.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 shutdown_connections 2024-01-21T11:20:31.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:31.249+0000 7f48baf90700 1 -- 172.21.15.178:0/3365791647 wait complete. 2024-01-21T11:20:31.250 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:32.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:31 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:31.248226+0000 mon.smithi089 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.178:0/3365791647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:32.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:32.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:33.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:31.706296+0000 mgr.smithi089.ufybqw (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 -- 172.21.15.178:0/2989876161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 msgr2=0x7f79740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 --2- 172.21.15.178:0/2989876161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f79740f55c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f7970009fb0 tx=0x7f7970009b30).stop 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 -- 172.21.15.178:0/2989876161 shutdown_connections 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 --2- 172.21.15.178:0/2989876161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f79740f55c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 -- 172.21.15.178:0/2989876161 >> 172.21.15.178:0/2989876161 conn(0x7f79740f09a0 msgr2=0x7f79740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 -- 172.21.15.178:0/2989876161 shutdown_connections 2024-01-21T11:20:33.747 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.747+0000 7f797ac97700 1 -- 172.21.15.178:0/2989876161 wait complete. 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f797ac97700 1 Processor -- start 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f797ac97700 1 -- start start 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f797ac97700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f7974109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f797ac97700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974060280 con 0x7f79740f51a0 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f7979c95700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f7974109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f7979c95700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f7974109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36070/0 (socket says 172.21.15.178:36070) 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f7979c95700 1 -- 172.21.15.178:0/954434924 learned_addr learned my addr 172.21.15.178:0/954434924 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.748+0000 7f7979c95700 1 -- 172.21.15.178:0/954434924 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7970005040 con 0x7f79740f51a0 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.749+0000 7f7979c95700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f7974109170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7970013510 tx=0x7f797000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:33.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.749+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7970009320 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.749+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7970013f00 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.749+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7974060480 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.749+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7970014dc0 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.749+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79741098a0 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.750+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79700095f0 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.750+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974045ad0 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.750+0000 7f796a7fc700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f796002ff60 0x7f7960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.750+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79700410c0 con 0x7f79740f51a0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.750+0000 7f7979494700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f796002ff60 0x7f7960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:33.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.750+0000 7f7979494700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f796002ff60 0x7f7960032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f79640096a0 tx=0x7f7964006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:33.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.754+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7970014420 con 0x7f79740f51a0 2024-01-21T11:20:33.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.960+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79740fabe0 con 0x7f79740f51a0 2024-01-21T11:20:33.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.961+0000 7f796a7fc700 1 -- 172.21.15.178:0/954434924 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7970010230 con 0x7f79740f51a0 2024-01-21T11:20:33.960 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:33.961 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f796002ff60 msgr2=0x7f7960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f796002ff60 0x7f7960032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f79640096a0 tx=0x7f7964006b40).stop 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 msgr2=0x7f7974109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f7974109170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7970013510 tx=0x7f797000b040).stop 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 shutdown_connections 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f796002ff60 0x7f7960032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 --2- 172.21.15.178:0/954434924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79740f51a0 0x7f7974109170 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 >> 172.21.15.178:0/954434924 conn(0x7f79740f09a0 msgr2=0x7f79740f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 shutdown_connections 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:33.962+0000 7f797ac97700 1 -- 172.21.15.178:0/954434924 wait complete. 2024-01-21T11:20:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:35.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:33.706648+0000 mgr.smithi089.ufybqw (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:35.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:33.961443+0000 mon.smithi089 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.178:0/954434924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:35.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:35.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 -- 172.21.15.178:0/796881227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 msgr2=0x7fd9500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 --2- 172.21.15.178:0/796881227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd9500f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fd940009fb0 tx=0x7fd940009b30).stop 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 -- 172.21.15.178:0/796881227 shutdown_connections 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 --2- 172.21.15.178:0/796881227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd9500f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 -- 172.21.15.178:0/796881227 >> 172.21.15.178:0/796881227 conn(0x7fd9500f1c70 msgr2=0x7fd9500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 -- 172.21.15.178:0/796881227 shutdown_connections 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.515+0000 7fd9579c5700 1 -- 172.21.15.178:0/796881227 wait complete. 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd9579c5700 1 Processor -- start 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd9579c5700 1 -- start start 2024-01-21T11:20:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd9579c5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd950106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd9579c5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950106c20 con 0x7fd9500f6510 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd955761700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd950106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd955761700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd950106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36076/0 (socket says 172.21.15.178:36076) 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd955761700 1 -- 172.21.15.178:0/3795243068 learned_addr learned my addr 172.21.15.178:0/3795243068 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.516+0000 7fd955761700 1 -- 172.21.15.178:0/3795243068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940005040 con 0x7fd9500f6510 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.517+0000 7fd955761700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd950106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd940013510 tx=0x7fd94000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.517+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd940009320 con 0x7fd9500f6510 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.517+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd950106e20 con 0x7fd9500f6510 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.517+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd940013f00 con 0x7fd9500f6510 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.517+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd940014d90 con 0x7fd9500f6510 2024-01-21T11:20:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.517+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9501072b0 con 0x7fd9500f6510 2024-01-21T11:20:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.518+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9400095f0 con 0x7fd9500f6510 2024-01-21T11:20:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.518+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950045ad0 con 0x7fd9500f6510 2024-01-21T11:20:36.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.518+0000 7fd945ffb700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd93c02ff60 0x7fd93c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:36.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.518+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd940041000 con 0x7fd9500f6510 2024-01-21T11:20:36.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.518+0000 7fd954f60700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd93c02ff60 0x7fd93c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:36.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.519+0000 7fd954f60700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd93c02ff60 0x7fd93c032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd94c0096a0 tx=0x7fd94c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:36.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.523+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd94001d7f0 con 0x7fd9500f6510 2024-01-21T11:20:36.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.732+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9500fbf60 con 0x7fd9500f6510 2024-01-21T11:20:36.734 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.733+0000 7fd945ffb700 1 -- 172.21.15.178:0/3795243068 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd940010230 con 0x7fd9500f6510 2024-01-21T11:20:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:36.734 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd93c02ff60 msgr2=0x7fd93c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd93c02ff60 0x7fd93c032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd94c0096a0 tx=0x7fd94c006b40).stop 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 msgr2=0x7fd950106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd950106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd940013510 tx=0x7fd94000b040).stop 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 shutdown_connections 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd93c02ff60 0x7fd93c032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 --2- 172.21.15.178:0/3795243068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9500f6510 0x7fd950106730 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 >> 172.21.15.178:0/3795243068 conn(0x7fd9500f1c70 msgr2=0x7fd9500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 shutdown_connections 2024-01-21T11:20:36.736 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:36.735+0000 7fd9579c5700 1 -- 172.21.15.178:0/3795243068 wait complete. 2024-01-21T11:20:36.737 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:37.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:35.707045+0000 mgr.smithi089.ufybqw (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:38.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:38.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:38.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:37 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:36.734744+0000 mon.smithi089 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.178:0/3795243068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:39.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:37.707291+0000 mgr.smithi089.ufybqw (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:39.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 -- 172.21.15.178:0/2512041616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 msgr2=0x7f10640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:39.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 --2- 172.21.15.178:0/2512041616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f10640f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f1054009fb0 tx=0x7f1054009b30).stop 2024-01-21T11:20:39.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 -- 172.21.15.178:0/2512041616 shutdown_connections 2024-01-21T11:20:39.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 --2- 172.21.15.178:0/2512041616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f10640f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:39.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 -- 172.21.15.178:0/2512041616 >> 172.21.15.178:0/2512041616 conn(0x7f10640f0b70 msgr2=0x7f10640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:39.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 -- 172.21.15.178:0/2512041616 shutdown_connections 2024-01-21T11:20:39.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 -- 172.21.15.178:0/2512041616 wait complete. 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.315+0000 7f1069910700 1 Processor -- start 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1069910700 1 -- start start 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1069910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f1064105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1069910700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1064105af0 con 0x7f10640f5410 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1062ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f1064105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1062ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f1064105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36104/0 (socket says 172.21.15.178:36104) 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1062ffd700 1 -- 172.21.15.178:0/624771933 learned_addr learned my addr 172.21.15.178:0/624771933 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.316+0000 7f1062ffd700 1 -- 172.21.15.178:0/624771933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1054005040 con 0x7f10640f5410 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.317+0000 7f1062ffd700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f1064105600 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f1054013510 tx=0x7f105400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.317+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1054009320 con 0x7f10640f5410 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.317+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1054013f00 con 0x7f10640f5410 2024-01-21T11:20:39.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.317+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064105cf0 con 0x7f10640f5410 2024-01-21T11:20:39.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.317+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1054014d90 con 0x7f10640f5410 2024-01-21T11:20:39.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.317+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1064106180 con 0x7f10640f5410 2024-01-21T11:20:39.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.318+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10540263a0 con 0x7f10640f5410 2024-01-21T11:20:39.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.318+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1064045ad0 con 0x7f10640f5410 2024-01-21T11:20:39.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.318+0000 7f104b7fe700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f104c02ff60 0x7f104c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:39.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.318+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1054041310 con 0x7f10640f5410 2024-01-21T11:20:39.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.318+0000 7f10627fc700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f104c02ff60 0x7f104c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:39.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.318+0000 7f10627fc700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f104c02ff60 0x7f104c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f10580096a0 tx=0x7f1058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:39.323 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.322+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f105401cb10 con 0x7f10640f5410 2024-01-21T11:20:39.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.533+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10640fae60 con 0x7f10640f5410 2024-01-21T11:20:39.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.533+0000 7f104b7fe700 1 -- 172.21.15.178:0/624771933 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f105401cb10 con 0x7f10640f5410 2024-01-21T11:20:39.535 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:39.535 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f104c02ff60 msgr2=0x7f104c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f104c02ff60 0x7f104c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f10580096a0 tx=0x7f1058006b40).stop 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 msgr2=0x7f1064105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f1064105600 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f1054013510 tx=0x7f105400b040).stop 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 shutdown_connections 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f104c02ff60 0x7f104c032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 --2- 172.21.15.178:0/624771933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10640f5410 0x7f1064105600 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 >> 172.21.15.178:0/624771933 conn(0x7f10640f0b70 msgr2=0x7f10640f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:39.536 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 shutdown_connections 2024-01-21T11:20:39.537 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:39.535+0000 7f1069910700 1 -- 172.21.15.178:0/624771933 wait complete. 2024-01-21T11:20:39.537 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:40.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:39.535241+0000 mon.smithi089 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.178:0/624771933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:40.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:40.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:41.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:39.707537+0000 mgr.smithi089.ufybqw (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:42.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- 172.21.15.178:0/1974014289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 msgr2=0x7f4ef40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:42.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 --2- 172.21.15.178:0/1974014289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef40f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f4ee4009fb0 tx=0x7f4ee4009b30).stop 2024-01-21T11:20:42.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- 172.21.15.178:0/1974014289 shutdown_connections 2024-01-21T11:20:42.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 --2- 172.21.15.178:0/1974014289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef40f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:42.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- 172.21.15.178:0/1974014289 >> 172.21.15.178:0/1974014289 conn(0x7f4ef40f0b70 msgr2=0x7f4ef40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- 172.21.15.178:0/1974014289 shutdown_connections 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- 172.21.15.178:0/1974014289 wait complete. 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 Processor -- start 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- start start 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4efbc7f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4060220 con 0x7f4ef40f5410 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.094+0000 7f4ef9a1b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4ef9a1b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:36126/0 (socket says 172.21.15.178:36126) 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4ef9a1b700 1 -- 172.21.15.178:0/2891015066 learned_addr learned my addr 172.21.15.178:0/2891015066 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4ef9a1b700 1 -- 172.21.15.178:0/2891015066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4005040 con 0x7f4ef40f5410 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4ef9a1b700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef4109450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4ee4013510 tx=0x7f4ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee4009320 con 0x7f4ef40f5410 2024-01-21T11:20:42.097 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef4060420 con 0x7f4ef40f5410 2024-01-21T11:20:42.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ee4013f00 con 0x7f4ef40f5410 2024-01-21T11:20:42.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee4014d90 con 0x7f4ef40f5410 2024-01-21T11:20:42.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.095+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef4109b00 con 0x7f4ef40f5410 2024-01-21T11:20:42.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.096+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ee40263a0 con 0x7f4ef40f5410 2024-01-21T11:20:42.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.096+0000 7f4eea7fc700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4ee002ff60 0x7f4ee0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:42.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.096+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ee40412f0 con 0x7f4ef40f5410 2024-01-21T11:20:42.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.096+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef4059100 con 0x7f4ef40f5410 2024-01-21T11:20:42.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.096+0000 7f4ef921a700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4ee002ff60 0x7f4ee0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:42.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.097+0000 7f4ef921a700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4ee002ff60 0x7f4ee0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f4ef00096a0 tx=0x7f4ef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:42.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.101+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ee4012920 con 0x7f4ef40f5410 2024-01-21T11:20:42.309 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.308+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ef410a1e0 con 0x7f4ef40f5410 2024-01-21T11:20:42.309 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.309+0000 7f4eea7fc700 1 -- 172.21.15.178:0/2891015066 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ee402b030 con 0x7f4ef40f5410 2024-01-21T11:20:42.310 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:42.310 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4ee002ff60 msgr2=0x7f4ee0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4ee002ff60 0x7f4ee0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f4ef00096a0 tx=0x7f4ef0006b40).stop 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 msgr2=0x7f4ef4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef4109450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4ee4013510 tx=0x7f4ee400b040).stop 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 shutdown_connections 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f4ee002ff60 0x7f4ee0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 --2- 172.21.15.178:0/2891015066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef40f5410 0x7f4ef4109450 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:42.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 >> 172.21.15.178:0/2891015066 conn(0x7f4ef40f0b70 msgr2=0x7f4ef40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:42.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 shutdown_connections 2024-01-21T11:20:42.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:42.310+0000 7f4efbc7f700 1 -- 172.21.15.178:0/2891015066 wait complete. 2024-01-21T11:20:42.312 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:43.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:41.707861+0000 mgr.smithi089.ufybqw (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:43.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:42.310501+0000 mon.smithi089 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.178:0/2891015066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:43.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:43.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:44.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/2038798625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 msgr2=0x7fe1e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:44.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 --2- 172.21.15.178:0/2038798625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e80f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fe1dc009fb0 tx=0x7fe1dc009b30).stop 2024-01-21T11:20:44.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/2038798625 shutdown_connections 2024-01-21T11:20:44.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 --2- 172.21.15.178:0/2038798625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e80f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:44.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/2038798625 >> 172.21.15.178:0/2038798625 conn(0x7fe1e80f0b70 msgr2=0x7fe1e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:44.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/2038798625 shutdown_connections 2024-01-21T11:20:44.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/2038798625 wait complete. 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 Processor -- start 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.855+0000 7fe1eeb8b700 1 -- start start 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.856+0000 7fe1eeb8b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.856+0000 7fe1eeb8b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8105b20 con 0x7fe1e80f5410 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.856+0000 7fe1ec927700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.856+0000 7fe1ec927700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47276/0 (socket says 172.21.15.178:47276) 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.856+0000 7fe1ec927700 1 -- 172.21.15.178:0/1432585830 learned_addr learned my addr 172.21.15.178:0/1432585830 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.856+0000 7fe1ec927700 1 -- 172.21.15.178:0/1432585830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc005040 con 0x7fe1e80f5410 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1ec927700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e8105630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe1dc013510 tx=0x7fe1dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1dc009320 con 0x7fe1e80f5410 2024-01-21T11:20:44.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e8105d20 con 0x7fe1e80f5410 2024-01-21T11:20:44.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1dc013f00 con 0x7fe1e80f5410 2024-01-21T11:20:44.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1dc014d90 con 0x7fe1e80f5410 2024-01-21T11:20:44.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e81061b0 con 0x7fe1e80f5410 2024-01-21T11:20:44.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.857+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1dc0095f0 con 0x7fe1e80f5410 2024-01-21T11:20:44.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.858+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e8045ad0 con 0x7fe1e80f5410 2024-01-21T11:20:44.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.858+0000 7fe1e57fa700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe1d802ff60 0x7fe1d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:44.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.858+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1dc040fe0 con 0x7fe1e80f5410 2024-01-21T11:20:44.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.858+0000 7fe1e7fff700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe1d802ff60 0x7fe1d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:44.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.858+0000 7fe1e7fff700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe1d802ff60 0x7fe1d8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe1d40096a0 tx=0x7fe1d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:44.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:44.863+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1dc01d7f0 con 0x7fe1e80f5410 2024-01-21T11:20:45.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.071+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1e80fae60 con 0x7fe1e80f5410 2024-01-21T11:20:45.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.072+0000 7fe1e57fa700 1 -- 172.21.15.178:0/1432585830 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe1dc010230 con 0x7fe1e80f5410 2024-01-21T11:20:45.072 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:45.073 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:45.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe1d802ff60 msgr2=0x7fe1d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:45.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe1d802ff60 0x7fe1d8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe1d40096a0 tx=0x7fe1d4006b40).stop 2024-01-21T11:20:45.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 msgr2=0x7fe1e8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:45.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e8105630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe1dc013510 tx=0x7fe1dc00b040).stop 2024-01-21T11:20:45.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 shutdown_connections 2024-01-21T11:20:45.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe1d802ff60 0x7fe1d8032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:45.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 --2- 172.21.15.178:0/1432585830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e80f5410 0x7fe1e8105630 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:45.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 >> 172.21.15.178:0/1432585830 conn(0x7fe1e80f0b70 msgr2=0x7fe1e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:45.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 shutdown_connections 2024-01-21T11:20:45.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:45.073+0000 7fe1eeb8b700 1 -- 172.21.15.178:0/1432585830 wait complete. 2024-01-21T11:20:45.075 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:45.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:43.708222+0000 mgr.smithi089.ufybqw (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:46.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:45.073304+0000 mon.smithi089 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.178:0/1432585830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:46.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:46.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:47.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:45.708592+0000 mgr.smithi089.ufybqw (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.620+0000 7ff058a0b700 1 -- 172.21.15.178:0/2207371740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 msgr2=0x7ff0540f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.620+0000 7ff058a0b700 1 --2- 172.21.15.178:0/2207371740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0540f66c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7ff048009fb0 tx=0x7ff048009b30).stop 2024-01-21T11:20:47.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.620+0000 7ff058a0b700 1 -- 172.21.15.178:0/2207371740 shutdown_connections 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.620+0000 7ff058a0b700 1 --2- 172.21.15.178:0/2207371740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0540f66c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.620+0000 7ff058a0b700 1 -- 172.21.15.178:0/2207371740 >> 172.21.15.178:0/2207371740 conn(0x7ff0540f1aa0 msgr2=0x7ff0540f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.621+0000 7ff058a0b700 1 -- 172.21.15.178:0/2207371740 shutdown_connections 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.621+0000 7ff058a0b700 1 -- 172.21.15.178:0/2207371740 wait complete. 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.621+0000 7ff058a0b700 1 Processor -- start 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.621+0000 7ff058a0b700 1 -- start start 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff058a0b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0541064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:47.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff058a0b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0541069a0 con 0x7ff0540f62a0 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff052d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0541064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff052d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0541064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47298/0 (socket says 172.21.15.178:47298) 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff052d9d700 1 -- 172.21.15.178:0/864902802 learned_addr learned my addr 172.21.15.178:0/864902802 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff052d9d700 1 -- 172.21.15.178:0/864902802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048005040 con 0x7ff0540f62a0 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.622+0000 7ff052d9d700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0541064b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff048009830 tx=0x7ff04800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff048009320 con 0x7ff0540f62a0 2024-01-21T11:20:47.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0480136c0 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff054106ba0 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff04801ee90 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff054107030 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff04801e420 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.623+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff054045ad0 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.624+0000 7ff03b7fe700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff03c02ff10 0x7ff03c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.624+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff048040f40 con 0x7ff0540f62a0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.624+0000 7ff05259c700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff03c02ff10 0x7ff03c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:47.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.624+0000 7ff05259c700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff03c02ff10 0x7ff03c0323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff0440096a0 tx=0x7ff044006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:47.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.628+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff04806d2b0 con 0x7ff0540f62a0 2024-01-21T11:20:47.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.833+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff054107e80 con 0x7ff0540f62a0 2024-01-21T11:20:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.834+0000 7ff03b7fe700 1 -- 172.21.15.178:0/864902802 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0480095f0 con 0x7ff0540f62a0 2024-01-21T11:20:47.835 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:47.836 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff03c02ff10 msgr2=0x7ff03c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff03c02ff10 0x7ff03c0323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff0440096a0 tx=0x7ff044006b40).stop 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 msgr2=0x7ff0541064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0541064b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff048009830 tx=0x7ff04800b040).stop 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 shutdown_connections 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff03c02ff10 0x7ff03c0323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 --2- 172.21.15.178:0/864902802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0540f62a0 0x7ff0541064b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 >> 172.21.15.178:0/864902802 conn(0x7ff0540f1aa0 msgr2=0x7ff0540f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 shutdown_connections 2024-01-21T11:20:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:47.836+0000 7ff058a0b700 1 -- 172.21.15.178:0/864902802 wait complete. 2024-01-21T11:20:47.838 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:48.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:47.708861+0000 mgr.smithi089.ufybqw (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:49.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:47.835974+0000 mon.smithi089 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.178:0/864902802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:49.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:49.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4089068476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 msgr2=0x7f2d5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 --2- 172.21.15.178:0/4089068476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c0f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f2d4c009fb0 tx=0x7f2d4c009b30).stop 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4089068476 shutdown_connections 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 --2- 172.21.15.178:0/4089068476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c0f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4089068476 >> 172.21.15.178:0/4089068476 conn(0x7f2d5c0f0b70 msgr2=0x7f2d5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4089068476 shutdown_connections 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.312+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4089068476 wait complete. 2024-01-21T11:20:50.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.313+0000 7f2d61b1f700 1 Processor -- start 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.313+0000 7f2d61b1f700 1 -- start start 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.313+0000 7f2d61b1f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.313+0000 7f2d61b1f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c060240 con 0x7f2d5c0f5410 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.313+0000 7f2d5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:47316/0 (socket says 172.21.15.178:47316) 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d5b7fe700 1 -- 172.21.15.178:0/4238864185 learned_addr learned my addr 172.21.15.178:0/4238864185 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d5b7fe700 1 -- 172.21.15.178:0/4238864185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c005040 con 0x7f2d5c0f5410 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d5b7fe700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c109430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f2d4c013510 tx=0x7f2d4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4c009320 con 0x7f2d5c0f5410 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c060440 con 0x7f2d5c0f5410 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d4c013f00 con 0x7f2d5c0f5410 2024-01-21T11:20:50.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4c014d90 con 0x7f2d5c0f5410 2024-01-21T11:20:50.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.314+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c109ae0 con 0x7f2d5c0f5410 2024-01-21T11:20:50.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.315+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d4c0263a0 con 0x7f2d5c0f5410 2024-01-21T11:20:50.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.315+0000 7f2d53fff700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d4802ff60 0x7f2d48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:50.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.315+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d4c040fd0 con 0x7f2d5c0f5410 2024-01-21T11:20:50.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.315+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c045ad0 con 0x7f2d5c0f5410 2024-01-21T11:20:50.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.315+0000 7f2d5affd700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d4802ff60 0x7f2d48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:50.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.316+0000 7f2d5affd700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d4802ff60 0x7f2d48032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2d440096a0 tx=0x7f2d44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:50.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.320+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d4c012920 con 0x7f2d5c0f5410 2024-01-21T11:20:50.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.525+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d5c0faf50 con 0x7f2d5c0f5410 2024-01-21T11:20:50.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.526+0000 7f2d53fff700 1 -- 172.21.15.178:0/4238864185 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d4c00f340 con 0x7f2d5c0f5410 2024-01-21T11:20:50.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:50.527 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:50.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.527+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d4802ff60 msgr2=0x7f2d48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:50.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.527+0000 7f2d61b1f700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d4802ff60 0x7f2d48032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2d440096a0 tx=0x7f2d44006b40).stop 2024-01-21T11:20:50.528 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.527+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 msgr2=0x7f2d5c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.527+0000 7f2d61b1f700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c109430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f2d4c013510 tx=0x7f2d4c00b040).stop 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.528+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 shutdown_connections 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.528+0000 7f2d61b1f700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d4802ff60 0x7f2d48032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.528+0000 7f2d61b1f700 1 --2- 172.21.15.178:0/4238864185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d5c0f5410 0x7f2d5c109430 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.528+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 >> 172.21.15.178:0/4238864185 conn(0x7f2d5c0f0b70 msgr2=0x7f2d5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.528+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 shutdown_connections 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:50.528+0000 7f2d61b1f700 1 -- 172.21.15.178:0/4238864185 wait complete. 2024-01-21T11:20:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:51.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:49.709112+0000 mgr.smithi089.ufybqw (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:51.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:50.527862+0000 mon.smithi089 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.178:0/4238864185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:51.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:51.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:53.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/2973211559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 msgr2=0x7f1cdc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:53.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 --2- 172.21.15.178:0/2973211559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc0f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f1ccc009fb0 tx=0x7f1ccc009b30).stop 2024-01-21T11:20:53.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/2973211559 shutdown_connections 2024-01-21T11:20:53.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 --2- 172.21.15.178:0/2973211559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc0f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:53.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/2973211559 >> 172.21.15.178:0/2973211559 conn(0x7f1cdc0f09a0 msgr2=0x7f1cdc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/2973211559 shutdown_connections 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/2973211559 wait complete. 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 Processor -- start 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- start start 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.129+0000 7f1ce2cdb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc060240 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce1cd9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce1cd9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55910/0 (socket says 172.21.15.178:55910) 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce1cd9700 1 -- 172.21.15.178:0/3132233996 learned_addr learned my addr 172.21.15.178:0/3132233996 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce1cd9700 1 -- 172.21.15.178:0/3132233996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc005040 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce1cd9700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc1091b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f1ccc013510 tx=0x7f1ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc009320 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc060440 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ccc013f00 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc014dc0 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.130+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc109860 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.131+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ccc01d3d0 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.131+0000 7f1cd27fc700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1cc802ffb0 0x7f1cc8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:53.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.131+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc045ad0 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.131+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ccc0413e0 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.132+0000 7f1ce14d8700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1cc802ffb0 0x7f1cc8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:53.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.132+0000 7f1ce14d8700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1cc802ffb0 0x7f1cc8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1cd80096a0 tx=0x7f1cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:53.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.136+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ccc014420 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:51.709401+0000 mgr.smithi089.ufybqw (mgr.14162) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:53.346 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.345+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cdc10a070 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.346+0000 7f1cd27fc700 1 -- 172.21.15.178:0/3132233996 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ccc01cb10 con 0x7f1cdc0f51a0 2024-01-21T11:20:53.374 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:53.374 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:53.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1cc802ffb0 msgr2=0x7f1cc8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:53.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1cc802ffb0 0x7f1cc8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1cd80096a0 tx=0x7f1cd8006b40).stop 2024-01-21T11:20:53.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 msgr2=0x7f1cdc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc1091b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f1ccc013510 tx=0x7f1ccc00b040).stop 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 shutdown_connections 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f1cc802ffb0 0x7f1cc8032470 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 --2- 172.21.15.178:0/3132233996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cdc0f51a0 0x7f1cdc1091b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 >> 172.21.15.178:0/3132233996 conn(0x7f1cdc0f09a0 msgr2=0x7f1cdc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 shutdown_connections 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:53.348+0000 7f1ce2cdb700 1 -- 172.21.15.178:0/3132233996 wait complete. 2024-01-21T11:20:53.376 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:54.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:53 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:53.347831+0000 mon.smithi089 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.178:0/3132233996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:54.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:54.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:55.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:53.709759+0000 mgr.smithi089.ufybqw (mgr.14162) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:55.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 -- 172.21.15.178:0/724999631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 msgr2=0x7ff7f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:55.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 --2- 172.21.15.178:0/724999631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f00f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7ff7d8009fb0 tx=0x7ff7d8009b30).stop 2024-01-21T11:20:55.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 -- 172.21.15.178:0/724999631 shutdown_connections 2024-01-21T11:20:55.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 --2- 172.21.15.178:0/724999631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f00f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:55.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 -- 172.21.15.178:0/724999631 >> 172.21.15.178:0/724999631 conn(0x7ff7f00f0b70 msgr2=0x7ff7f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:55.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 -- 172.21.15.178:0/724999631 shutdown_connections 2024-01-21T11:20:55.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.887+0000 7ff7f5172700 1 -- 172.21.15.178:0/724999631 wait complete. 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7f5172700 1 Processor -- start 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7f5172700 1 -- start start 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7f5172700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7f5172700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f0105b20 con 0x7ff7f00f5410 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7eed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7eed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55942/0 (socket says 172.21.15.178:55942) 2024-01-21T11:20:55.891 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.888+0000 7ff7eed9d700 1 -- 172.21.15.178:0/4135179271 learned_addr learned my addr 172.21.15.178:0/4135179271 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7eed9d700 1 -- 172.21.15.178:0/4135179271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d8005040 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7eed9d700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f0105630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ff7d8013510 tx=0x7ff7d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7d8009320 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f0105d20 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7d8013f00 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7d8014d90 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.889+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f01061b0 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.890+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7d8014420 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.890+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f0045ad0 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.890+0000 7ff7e77fe700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff7dc02ff60 0x7ff7dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.890+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7d8040fb0 con 0x7ff7f00f5410 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.890+0000 7ff7ee59c700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff7dc02ff60 0x7ff7dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:55.892 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.891+0000 7ff7ee59c700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff7dc02ff60 0x7ff7dc032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff7e00096a0 tx=0x7ff7e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:55.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:55.895+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7d80095f0 con 0x7ff7f00f5410 2024-01-21T11:20:56.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.104+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7f00fae60 con 0x7ff7f00f5410 2024-01-21T11:20:56.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.105+0000 7ff7e77fe700 1 -- 172.21.15.178:0/4135179271 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7d801db70 con 0x7ff7f00f5410 2024-01-21T11:20:56.106 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:56.106 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:56.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff7dc02ff60 msgr2=0x7ff7dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:56.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff7dc02ff60 0x7ff7dc032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff7e00096a0 tx=0x7ff7e0006b40).stop 2024-01-21T11:20:56.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 msgr2=0x7ff7f0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:56.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f0105630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ff7d8013510 tx=0x7ff7d800b040).stop 2024-01-21T11:20:56.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 shutdown_connections 2024-01-21T11:20:56.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7ff7dc02ff60 0x7ff7dc032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:56.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 --2- 172.21.15.178:0/4135179271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f00f5410 0x7ff7f0105630 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:56.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 >> 172.21.15.178:0/4135179271 conn(0x7ff7f00f0b70 msgr2=0x7ff7f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:56.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 shutdown_connections 2024-01-21T11:20:56.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:56.106+0000 7ff7f5172700 1 -- 172.21.15.178:0/4135179271 wait complete. 2024-01-21T11:20:56.108 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:57.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:55.710076+0000 mgr.smithi089.ufybqw (mgr.14162) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:20:57.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:56.106398+0000 mon.smithi089 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.178:0/4135179271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:20:57.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:20:57.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:20:58.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 -- 172.21.15.178:0/3199849376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 msgr2=0x7fc0400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 --2- 172.21.15.178:0/3199849376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc0400f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fc034009fb0 tx=0x7fc034009b30).stop 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 -- 172.21.15.178:0/3199849376 shutdown_connections 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 --2- 172.21.15.178:0/3199849376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc0400f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 -- 172.21.15.178:0/3199849376 >> 172.21.15.178:0/3199849376 conn(0x7fc0400f0b70 msgr2=0x7fc0400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 -- 172.21.15.178:0/3199849376 shutdown_connections 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.643+0000 7fc04807a700 1 -- 172.21.15.178:0/3199849376 wait complete. 2024-01-21T11:20:58.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc04807a700 1 Processor -- start 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc04807a700 1 -- start start 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc04807a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc040105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc04807a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040105b20 con 0x7fc0400f5410 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc045e16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc040105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc045e16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc040105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55966/0 (socket says 172.21.15.178:55966) 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc045e16700 1 -- 172.21.15.178:0/3041690436 learned_addr learned my addr 172.21.15.178:0/3041690436 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.644+0000 7fc045e16700 1 -- 172.21.15.178:0/3041690436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc034005040 con 0x7fc0400f5410 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.645+0000 7fc045e16700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc040105630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fc034014000 tx=0x7fc03400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.645+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0340095f0 con 0x7fc0400f5410 2024-01-21T11:20:58.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.645+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc040105d20 con 0x7fc0400f5410 2024-01-21T11:20:58.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.645+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc034007e70 con 0x7fc0400f5410 2024-01-21T11:20:58.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.645+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc034014ea0 con 0x7fc0400f5410 2024-01-21T11:20:58.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.645+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0401061b0 con 0x7fc0400f5410 2024-01-21T11:20:58.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.646+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc034027070 con 0x7fc0400f5410 2024-01-21T11:20:58.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.646+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc040045ad0 con 0x7fc0400f5410 2024-01-21T11:20:58.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.646+0000 7fc0327fc700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc02c02ff60 0x7fc02c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:20:58.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.646+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc034042790 con 0x7fc0400f5410 2024-01-21T11:20:58.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.646+0000 7fc045615700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc02c02ff60 0x7fc02c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:20:58.650 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.646+0000 7fc045615700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc02c02ff60 0x7fc02c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc03c0096a0 tx=0x7fc03c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:20:58.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.651+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0340274e0 con 0x7fc0400f5410 2024-01-21T11:20:58.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.855+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0400fae60 con 0x7fc0400f5410 2024-01-21T11:20:58.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc0327fc700 1 -- 172.21.15.178:0/3041690436 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc034027300 con 0x7fc0400f5410 2024-01-21T11:20:58.856 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:20:58.857 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:20:58.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc02c02ff60 msgr2=0x7fc02c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:58.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc02c02ff60 0x7fc02c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc03c0096a0 tx=0x7fc03c006b40).stop 2024-01-21T11:20:58.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 msgr2=0x7fc040105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc040105630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fc034014000 tx=0x7fc03400b040).stop 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 shutdown_connections 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc02c02ff60 0x7fc02c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 --2- 172.21.15.178:0/3041690436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0400f5410 0x7fc040105630 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 >> 172.21.15.178:0/3041690436 conn(0x7fc0400f0b70 msgr2=0x7fc0400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 shutdown_connections 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:20:58.857+0000 7fc04807a700 1 -- 172.21.15.178:0/3041690436 wait complete. 2024-01-21T11:20:58.859 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:20:59.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:57.710348+0000 mgr.smithi089.ufybqw (mgr.14162) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:00.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:20:59 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:20:58.857556+0000 mon.smithi089 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.178:0/3041690436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:00.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:00.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:01.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:20:59.710632+0000 mgr.smithi089.ufybqw (mgr.14162) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:01.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 -- 172.21.15.178:0/1390957968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 msgr2=0x7f2d480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:01.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 --2- 172.21.15.178:0/1390957968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d480f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f2d44009fb0 tx=0x7f2d44009b30).stop 2024-01-21T11:21:01.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 -- 172.21.15.178:0/1390957968 shutdown_connections 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 --2- 172.21.15.178:0/1390957968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d480f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 -- 172.21.15.178:0/1390957968 >> 172.21.15.178:0/1390957968 conn(0x7f2d480f0b70 msgr2=0x7f2d480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 -- 172.21.15.178:0/1390957968 shutdown_connections 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 -- 172.21.15.178:0/1390957968 wait complete. 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 Processor -- start 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.375+0000 7f2d50139700 1 -- start start 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d50139700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d48109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d50139700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d48060220 con 0x7f2d480f5410 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d4ded5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d48109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d4ded5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d48109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:55988/0 (socket says 172.21.15.178:55988) 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d4ded5700 1 -- 172.21.15.178:0/603768172 learned_addr learned my addr 172.21.15.178:0/603768172 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d4ded5700 1 -- 172.21.15.178:0/603768172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44005040 con 0x7f2d480f5410 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d4ded5700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d48109450 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f2d44013510 tx=0x7f2d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:01.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d44009320 con 0x7f2d480f5410 2024-01-21T11:21:01.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.376+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d48060420 con 0x7f2d480f5410 2024-01-21T11:21:01.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d44013f00 con 0x7f2d480f5410 2024-01-21T11:21:01.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d44014d90 con 0x7f2d480f5410 2024-01-21T11:21:01.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d48109b00 con 0x7f2d480f5410 2024-01-21T11:21:01.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d440095f0 con 0x7f2d480f5410 2024-01-21T11:21:01.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d48059100 con 0x7f2d480f5410 2024-01-21T11:21:01.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d3e7fc700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d3402ff60 0x7f2d34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:01.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.377+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d44041080 con 0x7f2d480f5410 2024-01-21T11:21:01.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.378+0000 7f2d4d6d4700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d3402ff60 0x7f2d34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:01.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.378+0000 7f2d4d6d4700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d3402ff60 0x7f2d34032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f2d380096a0 tx=0x7f2d38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:01.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.382+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d44014420 con 0x7f2d480f5410 2024-01-21T11:21:01.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.590+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d48045ad0 con 0x7f2d480f5410 2024-01-21T11:21:01.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.591+0000 7f2d3e7fc700 1 -- 172.21.15.178:0/603768172 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d44010230 con 0x7f2d480f5410 2024-01-21T11:21:01.591 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:01.591 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:01.592 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.592+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d3402ff60 msgr2=0x7f2d34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:01.592 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.592+0000 7f2d50139700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d3402ff60 0x7f2d34032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f2d380096a0 tx=0x7f2d38006b40).stop 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.592+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 msgr2=0x7f2d48109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.592+0000 7f2d50139700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d48109450 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f2d44013510 tx=0x7f2d4400b040).stop 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.593+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 shutdown_connections 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.593+0000 7f2d50139700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f2d3402ff60 0x7f2d34032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.593+0000 7f2d50139700 1 --2- 172.21.15.178:0/603768172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d480f5410 0x7f2d48109450 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.593+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 >> 172.21.15.178:0/603768172 conn(0x7f2d480f0b70 msgr2=0x7f2d480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.593+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 shutdown_connections 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:01.593+0000 7f2d50139700 1 -- 172.21.15.178:0/603768172 wait complete. 2024-01-21T11:21:01.593 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:02.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:01 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:01.591348+0000 mon.smithi089 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.178:0/603768172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:02.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:02.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:01.710944+0000 mgr.smithi089.ufybqw (mgr.14162) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:04.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 -- 172.21.15.178:0/3479385581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 msgr2=0x7f6b7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:04.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 --2- 172.21.15.178:0/3479385581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c0f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6b78009fb0 tx=0x7f6b78009b30).stop 2024-01-21T11:21:04.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 -- 172.21.15.178:0/3479385581 shutdown_connections 2024-01-21T11:21:04.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 --2- 172.21.15.178:0/3479385581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c0f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:04.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 -- 172.21.15.178:0/3479385581 >> 172.21.15.178:0/3479385581 conn(0x7f6b7c0f0b70 msgr2=0x7f6b7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 -- 172.21.15.178:0/3479385581 shutdown_connections 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.153+0000 7f6b844ec700 1 -- 172.21.15.178:0/3479385581 wait complete. 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b844ec700 1 Processor -- start 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b844ec700 1 -- start start 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b844ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b844ec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c060220 con 0x7f6b7c0f5410 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b82288700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b82288700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:33690/0 (socket says 172.21.15.178:33690) 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.154+0000 7f6b82288700 1 -- 172.21.15.178:0/3662408624 learned_addr learned my addr 172.21.15.178:0/3662408624 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b82288700 1 -- 172.21.15.178:0/3662408624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78005040 con 0x7f6b7c0f5410 2024-01-21T11:21:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b82288700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c109450 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6b78014000 tx=0x7f6b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:04.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b780095f0 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b78007e70 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c060420 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b78014ea0 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.155+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c109b00 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.156+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b7801d4e0 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.156+0000 7f6b72ffd700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6b6802ff60 0x7f6b68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.156+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c059100 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.156+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b78040880 con 0x7f6b7c0f5410 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.156+0000 7f6b81a87700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6b6802ff60 0x7f6b68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:04.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.156+0000 7f6b81a87700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6b6802ff60 0x7f6b68032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6b6c0096a0 tx=0x7f6b6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:04.160 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.160+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b7800f8f0 con 0x7f6b7c0f5410 2024-01-21T11:21:04.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.370+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b7c045ad0 con 0x7f6b7c0f5410 2024-01-21T11:21:04.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.371+0000 7f6b72ffd700 1 -- 172.21.15.178:0/3662408624 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b7802c090 con 0x7f6b7c0f5410 2024-01-21T11:21:04.371 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:04.371 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6b6802ff60 msgr2=0x7f6b68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6b6802ff60 0x7f6b68032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6b6c0096a0 tx=0x7f6b6c006b40).stop 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 msgr2=0x7f6b7c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c109450 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6b78014000 tx=0x7f6b7800b040).stop 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 shutdown_connections 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6b6802ff60 0x7f6b68032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 --2- 172.21.15.178:0/3662408624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b7c0f5410 0x7f6b7c109450 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 >> 172.21.15.178:0/3662408624 conn(0x7f6b7c0f0b70 msgr2=0x7f6b7c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 shutdown_connections 2024-01-21T11:21:04.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:04.373+0000 7f6b844ec700 1 -- 172.21.15.178:0/3662408624 wait complete. 2024-01-21T11:21:04.374 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:05.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:03.711261+0000 mgr.smithi089.ufybqw (mgr.14162) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:05.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:04.371849+0000 mon.smithi089 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.178:0/3662408624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:05.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:05.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.906+0000 7f167ffff700 1 -- 172.21.15.178:0/1960703881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 msgr2=0x7f16800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.906+0000 7f167ffff700 1 --2- 172.21.15.178:0/1960703881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f16800f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1670009fb0 tx=0x7f1670009b30).stop 2024-01-21T11:21:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.906+0000 7f167ffff700 1 -- 172.21.15.178:0/1960703881 shutdown_connections 2024-01-21T11:21:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.906+0000 7f167ffff700 1 --2- 172.21.15.178:0/1960703881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f16800f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.906+0000 7f167ffff700 1 -- 172.21.15.178:0/1960703881 >> 172.21.15.178:0/1960703881 conn(0x7f16800f09a0 msgr2=0x7f16800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.906+0000 7f167ffff700 1 -- 172.21.15.178:0/1960703881 shutdown_connections 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.907+0000 7f167ffff700 1 -- 172.21.15.178:0/1960703881 wait complete. 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.907+0000 7f167ffff700 1 Processor -- start 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.907+0000 7f167ffff700 1 -- start start 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.907+0000 7f167ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f1680109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.907+0000 7f167ffff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680060280 con 0x7f16800f51a0 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f1680109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f1680109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:33712/0 (socket says 172.21.15.178:33712) 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167effd700 1 -- 172.21.15.178:0/4291003288 learned_addr learned my addr 172.21.15.178:0/4291003288 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167effd700 1 -- 172.21.15.178:0/4291003288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1670005040 con 0x7f16800f51a0 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167effd700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f1680109190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f1670013510 tx=0x7f167000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1670009320 con 0x7f16800f51a0 2024-01-21T11:21:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1680060480 con 0x7f16800f51a0 2024-01-21T11:21:06.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1670013f00 con 0x7f16800f51a0 2024-01-21T11:21:06.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1670014dc0 con 0x7f16800f51a0 2024-01-21T11:21:06.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.908+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16801098c0 con 0x7f16800f51a0 2024-01-21T11:21:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.909+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16700095f0 con 0x7f16800f51a0 2024-01-21T11:21:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.909+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680045ad0 con 0x7f16800f51a0 2024-01-21T11:21:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.909+0000 7f16677fe700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f166802ff60 0x7f1668032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.909+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16700410c0 con 0x7f16800f51a0 2024-01-21T11:21:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.909+0000 7f167e7fc700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f166802ff60 0x7f1668032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.910+0000 7f167e7fc700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f166802ff60 0x7f1668032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f16740096a0 tx=0x7f1674006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:06.914+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f167001c790 con 0x7f16800f51a0 2024-01-21T11:21:07.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.118+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16800fabe0 con 0x7f16800f51a0 2024-01-21T11:21:07.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.119+0000 7f16677fe700 1 -- 172.21.15.178:0/4291003288 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1670010230 con 0x7f16800f51a0 2024-01-21T11:21:07.119 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:07.119 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:07.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f166802ff60 msgr2=0x7f1668032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:07.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f166802ff60 0x7f1668032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f16740096a0 tx=0x7f1674006b40).stop 2024-01-21T11:21:07.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 msgr2=0x7f1680109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:07.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f1680109190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f1670013510 tx=0x7f167000b040).stop 2024-01-21T11:21:07.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 shutdown_connections 2024-01-21T11:21:07.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f166802ff60 0x7f1668032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:07.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 --2- 172.21.15.178:0/4291003288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16800f51a0 0x7f1680109190 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:07.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 >> 172.21.15.178:0/4291003288 conn(0x7f16800f09a0 msgr2=0x7f16800f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:07.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 shutdown_connections 2024-01-21T11:21:07.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:07.120+0000 7f167ffff700 1 -- 172.21.15.178:0/4291003288 wait complete. 2024-01-21T11:21:07.121 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:07.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:05.711624+0000 mgr.smithi089.ufybqw (mgr.14162) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:08.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:07 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:07.119683+0000 mon.smithi089 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.178:0/4291003288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:08.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:08.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:09.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:07.711949+0000 mgr.smithi089.ufybqw (mgr.14162) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 -- 172.21.15.178:0/3767839227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 msgr2=0x7fec680fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 --2- 172.21.15.178:0/3767839227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec680fb380 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fec58009fb0 tx=0x7fec58009b30).stop 2024-01-21T11:21:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 -- 172.21.15.178:0/3767839227 shutdown_connections 2024-01-21T11:21:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 --2- 172.21.15.178:0/3767839227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec680fb380 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 -- 172.21.15.178:0/3767839227 >> 172.21.15.178:0/3767839227 conn(0x7fec680f09a0 msgr2=0x7fec680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:09.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 -- 172.21.15.178:0/3767839227 shutdown_connections 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 -- 172.21.15.178:0/3767839227 wait complete. 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 Processor -- start 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.650+0000 7fec6f510700 1 -- start start 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6f510700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec681086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6f510700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68108bc0 con 0x7fec68060180 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6e50e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec681086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6e50e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec681086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:33736/0 (socket says 172.21.15.178:33736) 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6e50e700 1 -- 172.21.15.178:0/3307912335 learned_addr learned my addr 172.21.15.178:0/3307912335 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6e50e700 1 -- 172.21.15.178:0/3307912335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58005040 con 0x7fec68060180 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6e50e700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec681086d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fec58013510 tx=0x7fec5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec58009320 con 0x7fec68060180 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6f510700 1 -- 172.21.15.178:0/3307912335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec68108dc0 con 0x7fec68060180 2024-01-21T11:21:09.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec6f510700 1 -- 172.21.15.178:0/3307912335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec68109250 con 0x7fec68060180 2024-01-21T11:21:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.651+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec58013f00 con 0x7fec68060180 2024-01-21T11:21:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.652+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec58014d90 con 0x7fec68060180 2024-01-21T11:21:09.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.652+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec68045ad0 con 0x7fec68060180 2024-01-21T11:21:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.653+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec580095f0 con 0x7fec68060180 2024-01-21T11:21:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.653+0000 7fec5effd700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fec5402ffb0 0x7fec54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.653+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec58040f80 con 0x7fec68060180 2024-01-21T11:21:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.653+0000 7fec6dd0d700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fec5402ffb0 0x7fec54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:09.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.653+0000 7fec6dd0d700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fec5402ffb0 0x7fec54032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fec640096a0 tx=0x7fec64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:09.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.657+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec58014420 con 0x7fec68060180 2024-01-21T11:21:09.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.861+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec680fdee0 con 0x7fec68060180 2024-01-21T11:21:09.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.861+0000 7fec5effd700 1 -- 172.21.15.178:0/3307912335 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec58010230 con 0x7fec68060180 2024-01-21T11:21:09.861 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:09.861 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fec5402ffb0 msgr2=0x7fec54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fec5402ffb0 0x7fec54032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fec640096a0 tx=0x7fec64006b40).stop 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 msgr2=0x7fec681086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec681086d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fec58013510 tx=0x7fec5800b040).stop 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 shutdown_connections 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fec5402ffb0 0x7fec54032470 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:09.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 --2- 172.21.15.178:0/3307912335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec68060180 0x7fec681086d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:09.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 >> 172.21.15.178:0/3307912335 conn(0x7fec680f09a0 msgr2=0x7fec680f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:09.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 shutdown_connections 2024-01-21T11:21:09.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:09.863+0000 7fec5cff9700 1 -- 172.21.15.178:0/3307912335 wait complete. 2024-01-21T11:21:09.864 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:11.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:09.712177+0000 mgr.smithi089.ufybqw (mgr.14162) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:11.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:09.862161+0000 mon.smithi089 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.178:0/3307912335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:11.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:11.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:12.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:11 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:11.712538+0000 mgr.smithi089.ufybqw (mgr.14162) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.347+0000 7f77abfff700 1 -- 172.21.15.178:0/459232623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 msgr2=0x7f77ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.347+0000 7f77abfff700 1 --2- 172.21.15.178:0/459232623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac0f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f779c009fb0 tx=0x7f779c009b30).stop 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.347+0000 7f77abfff700 1 -- 172.21.15.178:0/459232623 shutdown_connections 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.347+0000 7f77abfff700 1 --2- 172.21.15.178:0/459232623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac0f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.347+0000 7f77abfff700 1 -- 172.21.15.178:0/459232623 >> 172.21.15.178:0/459232623 conn(0x7f77ac0f09a0 msgr2=0x7f77ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.348+0000 7f77abfff700 1 -- 172.21.15.178:0/459232623 shutdown_connections 2024-01-21T11:21:12.348 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.348+0000 7f77abfff700 1 -- 172.21.15.178:0/459232623 wait complete. 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.348+0000 7f77abfff700 1 Processor -- start 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.348+0000 7f77abfff700 1 -- start start 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77abfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77abfff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac060280 con 0x7f77ac0f51a0 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52906/0 (socket says 172.21.15.178:52906) 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77aaffd700 1 -- 172.21.15.178:0/518651755 learned_addr learned my addr 172.21.15.178:0/518651755 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77aaffd700 1 -- 172.21.15.178:0/518651755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c005040 con 0x7f77ac0f51a0 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77aaffd700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac109170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f779c013510 tx=0x7f779c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.349+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f779c009320 con 0x7f77ac0f51a0 2024-01-21T11:21:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.350+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f779c013f00 con 0x7f77ac0f51a0 2024-01-21T11:21:12.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.350+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac060480 con 0x7f77ac0f51a0 2024-01-21T11:21:12.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.350+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f779c014dc0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.350+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac1098a0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.350+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f779c0095f0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.351+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac045ad0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.351+0000 7f77937fe700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f779402ffb0 0x7f7794032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:12.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.351+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f779c040fe0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.351+0000 7f77aa7fc700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f779402ffb0 0x7f7794032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:12.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.351+0000 7f77aa7fc700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f779402ffb0 0x7f7794032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f77a00096a0 tx=0x7f77a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:12.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.356+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f779c06d2b0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.561 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.561+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77ac0fabe0 con 0x7f77ac0f51a0 2024-01-21T11:21:12.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.562+0000 7f77937fe700 1 -- 172.21.15.178:0/518651755 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f779c014420 con 0x7f77ac0f51a0 2024-01-21T11:21:12.562 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:12.562 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:12.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f779402ffb0 msgr2=0x7f7794032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f779402ffb0 0x7f7794032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f77a00096a0 tx=0x7f77a0006b40).stop 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 msgr2=0x7f77ac109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac109170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f779c013510 tx=0x7f779c00b040).stop 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 shutdown_connections 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f779402ffb0 0x7f7794032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 --2- 172.21.15.178:0/518651755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0f51a0 0x7f77ac109170 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 >> 172.21.15.178:0/518651755 conn(0x7f77ac0f09a0 msgr2=0x7f77ac0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 shutdown_connections 2024-01-21T11:21:12.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:12.564+0000 7f77abfff700 1 -- 172.21.15.178:0/518651755 wait complete. 2024-01-21T11:21:12.565 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:13.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:12.562939+0000 mon.smithi089 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.178:0/518651755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:13.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:13.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:14.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:13.712861+0000 mgr.smithi089.ufybqw (mgr.14162) 246 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:14.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:13.778950+0000 mon.smithi089 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:21:14.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:13.787263+0000 mon.smithi089 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.89:0/134633192' entity='mgr.smithi089.ufybqw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.ufybqw/trash_purge_schedule"}]: dispatch 2024-01-21T11:21:15.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 -- 172.21.15.178:0/3703752936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 msgr2=0x7fd90c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 --2- 172.21.15.178:0/3703752936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c0f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fd900009fb0 tx=0x7fd900009b30).stop 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 -- 172.21.15.178:0/3703752936 shutdown_connections 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 --2- 172.21.15.178:0/3703752936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c0f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 -- 172.21.15.178:0/3703752936 >> 172.21.15.178:0/3703752936 conn(0x7fd90c0f0b70 msgr2=0x7fd90c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 -- 172.21.15.178:0/3703752936 shutdown_connections 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 -- 172.21.15.178:0/3703752936 wait complete. 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 Processor -- start 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.056+0000 7fd9121eb700 1 -- start start 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd9121eb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd9121eb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c105b20 con 0x7fd90c0f5410 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52932/0 (socket says 172.21.15.178:52932) 2024-01-21T11:21:15.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd90b7fe700 1 -- 172.21.15.178:0/1008923078 learned_addr learned my addr 172.21.15.178:0/1008923078 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd90b7fe700 1 -- 172.21.15.178:0/1008923078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900005040 con 0x7fd90c0f5410 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.057+0000 7fd90b7fe700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd900013510 tx=0x7fd90000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900009320 con 0x7fd90c0f5410 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c105d20 con 0x7fd90c0f5410 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd900013f00 con 0x7fd90c0f5410 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900014d90 con 0x7fd90c0f5410 2024-01-21T11:21:15.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c1061b0 con 0x7fd90c0f5410 2024-01-21T11:21:15.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9000095f0 con 0x7fd90c0f5410 2024-01-21T11:21:15.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.058+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c045ad0 con 0x7fd90c0f5410 2024-01-21T11:21:15.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.059+0000 7fd8f3fff700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd8f402ff60 0x7fd8f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:15.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.059+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd900041000 con 0x7fd90c0f5410 2024-01-21T11:21:15.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.059+0000 7fd90affd700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd8f402ff60 0x7fd8f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:15.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.059+0000 7fd90affd700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd8f402ff60 0x7fd8f4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd8fc0096a0 tx=0x7fd8fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:15.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.063+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd900014420 con 0x7fd90c0f5410 2024-01-21T11:21:15.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.272+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd90c0fae60 con 0x7fd90c0f5410 2024-01-21T11:21:15.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.273+0000 7fd8f3fff700 1 -- 172.21.15.178:0/1008923078 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd900010230 con 0x7fd90c0f5410 2024-01-21T11:21:15.273 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:15.273 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:15.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd8f402ff60 msgr2=0x7fd8f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:15.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd8f402ff60 0x7fd8f4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd8fc0096a0 tx=0x7fd8fc006b40).stop 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 msgr2=0x7fd90c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd900013510 tx=0x7fd90000b040).stop 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 shutdown_connections 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fd8f402ff60 0x7fd8f4032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 --2- 172.21.15.178:0/1008923078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0f5410 0x7fd90c105630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.274+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 >> 172.21.15.178:0/1008923078 conn(0x7fd90c0f0b70 msgr2=0x7fd90c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.275+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 shutdown_connections 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:15.275+0000 7fd9121eb700 1 -- 172.21.15.178:0/1008923078 wait complete. 2024-01-21T11:21:15.275 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:16.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:15 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:15.273617+0000 mon.smithi089 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.178:0/1008923078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:16.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:16.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:17.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:16 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:15.713245+0000 mgr.smithi089.ufybqw (mgr.14162) 247 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.829+0000 7f005fb85700 1 -- 172.21.15.178:0/493842658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 msgr2=0x7f00580f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.829+0000 7f005fb85700 1 --2- 172.21.15.178:0/493842658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00580f5df0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f0048009fb0 tx=0x7f0048009b30).stop 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.829+0000 7f005fb85700 1 -- 172.21.15.178:0/493842658 shutdown_connections 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.829+0000 7f005fb85700 1 --2- 172.21.15.178:0/493842658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00580f5df0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.829+0000 7f005fb85700 1 -- 172.21.15.178:0/493842658 >> 172.21.15.178:0/493842658 conn(0x7f00580f02e0 msgr2=0x7f00580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005fb85700 1 -- 172.21.15.178:0/493842658 shutdown_connections 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005fb85700 1 -- 172.21.15.178:0/493842658 wait complete. 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005fb85700 1 Processor -- start 2024-01-21T11:21:17.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005fb85700 1 -- start start 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005fb85700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00581093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005fb85700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580f5390 con 0x7f00580f59d0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005d921700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00581093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005d921700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00581093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52940/0 (socket says 172.21.15.178:52940) 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.830+0000 7f005d921700 1 -- 172.21.15.178:0/72335916 learned_addr learned my addr 172.21.15.178:0/72335916 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f005d921700 1 -- 172.21.15.178:0/72335916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0048005040 con 0x7f00580f59d0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f005d921700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00581093e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0048013510 tx=0x7f004800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048009320 con 0x7f00580f59d0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00580f5590 con 0x7f00580f59d0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0048013f00 con 0x7f00580f59d0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048014d90 con 0x7f00580f59d0 2024-01-21T11:21:17.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.831+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0058109a90 con 0x7f00580f59d0 2024-01-21T11:21:17.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.832+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00480095f0 con 0x7f00580f59d0 2024-01-21T11:21:17.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.832+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0058059100 con 0x7f00580f59d0 2024-01-21T11:21:17.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.832+0000 7f004e7fc700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f004402ff60 0x7f0044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:17.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.832+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0048041080 con 0x7f00580f59d0 2024-01-21T11:21:17.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.832+0000 7f005d120700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f004402ff60 0x7f0044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:17.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.833+0000 7f005d120700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f004402ff60 0x7f0044032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f00540096a0 tx=0x7f0054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:17.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:17.837+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0048014420 con 0x7f00580f59d0 2024-01-21T11:21:18.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.042+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00580fae50 con 0x7f00580f59d0 2024-01-21T11:21:18.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.043+0000 7f004e7fc700 1 -- 172.21.15.178:0/72335916 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0048010230 con 0x7f00580f59d0 2024-01-21T11:21:18.043 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:18.043 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:18.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.044+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f004402ff60 msgr2=0x7f0044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:18.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.044+0000 7f005fb85700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f004402ff60 0x7f0044032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f00540096a0 tx=0x7f0054006b40).stop 2024-01-21T11:21:18.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.044+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 msgr2=0x7f00581093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:18.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.044+0000 7f005fb85700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00581093e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0048013510 tx=0x7f004800b040).stop 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.044+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 shutdown_connections 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.045+0000 7f005fb85700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f004402ff60 0x7f0044032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.045+0000 7f005fb85700 1 --2- 172.21.15.178:0/72335916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00580f59d0 0x7f00581093e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.045+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 >> 172.21.15.178:0/72335916 conn(0x7f00580f02e0 msgr2=0x7f00580f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.045+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 shutdown_connections 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:18.045+0000 7f005fb85700 1 -- 172.21.15.178:0/72335916 wait complete. 2024-01-21T11:21:18.045 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:19.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:17.713588+0000 mgr.smithi089.ufybqw (mgr.14162) 248 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:19.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:18 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:18.043731+0000 mon.smithi089 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.178:0/72335916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:19.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:19.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:20.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 -- 172.21.15.178:0/1853378813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 msgr2=0x7fa94c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:20.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 --2- 172.21.15.178:0/1853378813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0f55c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fa93c009fb0 tx=0x7fa93c009b30).stop 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 -- 172.21.15.178:0/1853378813 shutdown_connections 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 --2- 172.21.15.178:0/1853378813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0f55c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 -- 172.21.15.178:0/1853378813 >> 172.21.15.178:0/1853378813 conn(0x7fa94c0f09a0 msgr2=0x7fa94c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 -- 172.21.15.178:0/1853378813 shutdown_connections 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 -- 172.21.15.178:0/1853378813 wait complete. 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.566+0000 7fa94bfff700 1 Processor -- start 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94bfff700 1 -- start start 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94bfff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c0fb5a0 con 0x7fa94c0f51a0 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52960/0 (socket says 172.21.15.178:52960) 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94affd700 1 -- 172.21.15.178:0/1744257959 learned_addr learned my addr 172.21.15.178:0/1744257959 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:20.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94affd700 1 -- 172.21.15.178:0/1744257959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c005040 con 0x7fa94c0f51a0 2024-01-21T11:21:20.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.567+0000 7fa94affd700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0fd120 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa93c013510 tx=0x7fa93c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:20.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.568+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c009320 con 0x7fa94c0f51a0 2024-01-21T11:21:20.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.568+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa93c013f00 con 0x7fa94c0f51a0 2024-01-21T11:21:20.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.568+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c0fb740 con 0x7fa94c0f51a0 2024-01-21T11:21:20.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.568+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c014dc0 con 0x7fa94c0f51a0 2024-01-21T11:21:20.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.568+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c0fbbd0 con 0x7fa94c0f51a0 2024-01-21T11:21:20.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.568+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa93c0095f0 con 0x7fa94c0f51a0 2024-01-21T11:21:20.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.569+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c059100 con 0x7fa94c0f51a0 2024-01-21T11:21:20.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.569+0000 7fa9337fe700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa93402ff60 0x7fa934032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:20.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.569+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa93c041100 con 0x7fa94c0f51a0 2024-01-21T11:21:20.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.569+0000 7fa94a7fc700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa93402ff60 0x7fa934032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:20.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.569+0000 7fa94a7fc700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa93402ff60 0x7fa934032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa9400096a0 tx=0x7fa940006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:20.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.573+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa93c014420 con 0x7fa94c0f51a0 2024-01-21T11:21:20.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.779+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa94c0fabe0 con 0x7fa94c0f51a0 2024-01-21T11:21:20.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.780+0000 7fa9337fe700 1 -- 172.21.15.178:0/1744257959 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa93c010230 con 0x7fa94c0f51a0 2024-01-21T11:21:20.780 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:20.780 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:20.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa93402ff60 msgr2=0x7fa934032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:20.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa93402ff60 0x7fa934032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa9400096a0 tx=0x7fa940006b40).stop 2024-01-21T11:21:20.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 msgr2=0x7fa94c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0fd120 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa93c013510 tx=0x7fa93c00b040).stop 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 shutdown_connections 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa93402ff60 0x7fa934032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 --2- 172.21.15.178:0/1744257959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0fd120 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.781+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 >> 172.21.15.178:0/1744257959 conn(0x7fa94c0f09a0 msgr2=0x7fa94c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.782+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 shutdown_connections 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:20.782+0000 7fa94bfff700 1 -- 172.21.15.178:0/1744257959 wait complete. 2024-01-21T11:21:20.782 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:21.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:20 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:19.713842+0000 mgr.smithi089.ufybqw (mgr.14162) 249 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:22.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:22.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:22.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:21 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:20.780775+0000 mon.smithi089 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.178:0/1744257959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:23.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:22 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:21.714087+0000 mgr.smithi089.ufybqw (mgr.14162) 250 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:23.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 -- 172.21.15.178:0/3550683975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 msgr2=0x7f8c140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:23.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 --2- 172.21.15.178:0/3550683975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f8c08009fb0 tx=0x7f8c08009b30).stop 2024-01-21T11:21:23.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 -- 172.21.15.178:0/3550683975 shutdown_connections 2024-01-21T11:21:23.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 --2- 172.21.15.178:0/3550683975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:23.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 -- 172.21.15.178:0/3550683975 >> 172.21.15.178:0/3550683975 conn(0x7f8c140f0b70 msgr2=0x7f8c140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 -- 172.21.15.178:0/3550683975 shutdown_connections 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.298+0000 7f8c19a08700 1 -- 172.21.15.178:0/3550683975 wait complete. 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c19a08700 1 Processor -- start 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c19a08700 1 -- start start 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c19a08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c19a08700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140f9a00 con 0x7f8c140f5410 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c12ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c12ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41850/0 (socket says 172.21.15.178:41850) 2024-01-21T11:21:23.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.299+0000 7f8c12ffd700 1 -- 172.21.15.178:0/3073920254 learned_addr learned my addr 172.21.15.178:0/3073920254 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8c12ffd700 1 -- 172.21.15.178:0/3073920254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c08005040 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8c12ffd700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140fb580 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f8c08013510 tx=0x7f8c0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c08009320 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c140f9ba0 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c08013f00 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c08014d90 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.300+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c140fa030 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.301+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c08014420 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.301+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14059100 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.301+0000 7f8bfb7fe700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8bfc02ff60 0x7f8bfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.301+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c080410d0 con 0x7f8c140f5410 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.301+0000 7f8c127fc700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8bfc02ff60 0x7f8bfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:23.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.302+0000 7f8c127fc700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8bfc02ff60 0x7f8bfc032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8c040096a0 tx=0x7f8c04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:23.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.306+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c080095f0 con 0x7f8c140f5410 2024-01-21T11:21:23.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.511+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c140fbf90 con 0x7f8c140f5410 2024-01-21T11:21:23.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.512+0000 7f8bfb7fe700 1 -- 172.21.15.178:0/3073920254 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c0801db70 con 0x7f8c140f5410 2024-01-21T11:21:23.512 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:23.512 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8bfc02ff60 msgr2=0x7f8bfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8bfc02ff60 0x7f8bfc032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8c040096a0 tx=0x7f8c04006b40).stop 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 msgr2=0x7f8c140fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140fb580 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f8c08013510 tx=0x7f8c0800b040).stop 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 shutdown_connections 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8bfc02ff60 0x7f8bfc032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 --2- 172.21.15.178:0/3073920254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c140f5410 0x7f8c140fb580 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 >> 172.21.15.178:0/3073920254 conn(0x7f8c140f0b70 msgr2=0x7f8c140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 shutdown_connections 2024-01-21T11:21:23.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:23.514+0000 7f8c19a08700 1 -- 172.21.15.178:0/3073920254 wait complete. 2024-01-21T11:21:23.515 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:24.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:23 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:23.512978+0000 mon.smithi089 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.178:0/3073920254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:24.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:24.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:25.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:24 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:23.714397+0000 mgr.smithi089.ufybqw (mgr.14162) 251 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 -- 172.21.15.178:0/1931625645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 msgr2=0x7fc0fc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 --2- 172.21.15.178:0/1931625645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc0f4be0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fc0ec009fb0 tx=0x7fc0ec009b30).stop 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 -- 172.21.15.178:0/1931625645 shutdown_connections 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 --2- 172.21.15.178:0/1931625645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc0f4be0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 -- 172.21.15.178:0/1931625645 >> 172.21.15.178:0/1931625645 conn(0x7fc0fc0f02e0 msgr2=0x7fc0fc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 -- 172.21.15.178:0/1931625645 shutdown_connections 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.082+0000 7fc101957700 1 -- 172.21.15.178:0/1931625645 wait complete. 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc101957700 1 Processor -- start 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc101957700 1 -- start start 2024-01-21T11:21:26.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc101957700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc101957700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc106bf0 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc0faffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc0faffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41878/0 (socket says 172.21.15.178:41878) 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc0faffd700 1 -- 172.21.15.178:0/3456677612 learned_addr learned my addr 172.21.15.178:0/3456677612 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.083+0000 7fc0faffd700 1 -- 172.21.15.178:0/3456677612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec005040 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.084+0000 7fc0faffd700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc106700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fc0ec013510 tx=0x7fc0ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.084+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0ec009320 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.084+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0ec013f00 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.084+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0ec014d90 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.084+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc106df0 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.084+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc107280 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.085+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0ec0095f0 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.085+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0fc045ad0 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.085+0000 7fc0e37fe700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc0e402ffb0 0x7fc0e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:26.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.085+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0ec040ff0 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.085+0000 7fc0fa7fc700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc0e402ffb0 0x7fc0e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:26.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.085+0000 7fc0fa7fc700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc0e402ffb0 0x7fc0e4032470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc0f00096a0 tx=0x7fc0f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:26.089 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.090+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0ec06d2b0 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.294 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.295+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0fc108130 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.295+0000 7fc0e37fe700 1 -- 172.21.15.178:0/3456677612 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0ec014420 con 0x7fc0fc0f47c0 2024-01-21T11:21:26.295 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:26.296 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:26.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc0e402ffb0 msgr2=0x7fc0e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:26.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc0e402ffb0 0x7fc0e4032470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc0f00096a0 tx=0x7fc0f0006b40).stop 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 msgr2=0x7fc0fc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc106700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fc0ec013510 tx=0x7fc0ec00b040).stop 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 shutdown_connections 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fc0e402ffb0 0x7fc0e4032470 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 --2- 172.21.15.178:0/3456677612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc0f47c0 0x7fc0fc106700 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 >> 172.21.15.178:0/3456677612 conn(0x7fc0fc0f02e0 msgr2=0x7fc0fc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 shutdown_connections 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:26.297+0000 7fc101957700 1 -- 172.21.15.178:0/3456677612 wait complete. 2024-01-21T11:21:26.298 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:27.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:25.714721+0000 mgr.smithi089.ufybqw (mgr.14162) 252 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:27.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:26 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:26.296094+0000 mon.smithi089 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.178:0/3456677612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:27.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:27.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.822+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3305208858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 msgr2=0x7fbdf00f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.822+0000 7fbdf7e5b700 1 --2- 172.21.15.178:0/3305208858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf00f68a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbde0009fb0 tx=0x7fbde0009b30).stop 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.822+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3305208858 shutdown_connections 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.822+0000 7fbdf7e5b700 1 --2- 172.21.15.178:0/3305208858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf00f68a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.822+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3305208858 >> 172.21.15.178:0/3305208858 conn(0x7fbdf00f1c00 msgr2=0x7fbdf00f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3305208858 shutdown_connections 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3305208858 wait complete. 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf7e5b700 1 Processor -- start 2024-01-21T11:21:28.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf7e5b700 1 -- start start 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf7e5b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf7e5b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0106bc0 con 0x7fbdf00f6480 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf5bf7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf5bf7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41894/0 (socket says 172.21.15.178:41894) 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.823+0000 7fbdf5bf7700 1 -- 172.21.15.178:0/3778087928 learned_addr learned my addr 172.21.15.178:0/3778087928 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbdf5bf7700 1 -- 172.21.15.178:0/3778087928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde0005040 con 0x7fbdf00f6480 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbdf5bf7700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf01066d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fbde0013510 tx=0x7fbde000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbde0009320 con 0x7fbdf00f6480 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf0106dc0 con 0x7fbdf00f6480 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbde0013f00 con 0x7fbdf00f6480 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbde0014d90 con 0x7fbdf00f6480 2024-01-21T11:21:28.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.824+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf0107250 con 0x7fbdf00f6480 2024-01-21T11:21:28.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.825+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf0045ad0 con 0x7fbdf00f6480 2024-01-21T11:21:28.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.825+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbde00095f0 con 0x7fbdf00f6480 2024-01-21T11:21:28.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.825+0000 7fbde67fc700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbddc02ff60 0x7fbddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:28.827 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.825+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbde0040fe0 con 0x7fbdf00f6480 2024-01-21T11:21:28.827 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.825+0000 7fbdf53f6700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbddc02ff60 0x7fbddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:28.827 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.826+0000 7fbdf53f6700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbddc02ff60 0x7fbddc032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbdec0096a0 tx=0x7fbdec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:28.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:28.830+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbde0010230 con 0x7fbdf00f6480 2024-01-21T11:21:29.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.035+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdf01080a0 con 0x7fbdf00f6480 2024-01-21T11:21:29.036 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.036+0000 7fbde67fc700 1 -- 172.21.15.178:0/3778087928 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbde0010230 con 0x7fbdf00f6480 2024-01-21T11:21:29.036 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:29.036 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:29.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbddc02ff60 msgr2=0x7fbddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbddc02ff60 0x7fbddc032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbdec0096a0 tx=0x7fbdec006b40).stop 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 msgr2=0x7fbdf01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf01066d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fbde0013510 tx=0x7fbde000b040).stop 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 shutdown_connections 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fbddc02ff60 0x7fbddc032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 --2- 172.21.15.178:0/3778087928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdf00f6480 0x7fbdf01066d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 >> 172.21.15.178:0/3778087928 conn(0x7fbdf00f1c00 msgr2=0x7fbdf00f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 shutdown_connections 2024-01-21T11:21:29.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:29.038+0000 7fbdf7e5b700 1 -- 172.21.15.178:0/3778087928 wait complete. 2024-01-21T11:21:29.039 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:29.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:28 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:27.715014+0000 mgr.smithi089.ufybqw (mgr.14162) 253 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:30.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:29 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:29.036833+0000 mon.smithi089 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.178:0/3778087928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:30.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:30.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:31.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:30 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:29.715338+0000 mgr.smithi089.ufybqw (mgr.14162) 254 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 -- 172.21.15.178:0/1688842984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 msgr2=0x7f37880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 --2- 172.21.15.178:0/1688842984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f37880f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f3778009fb0 tx=0x7f3778009b30).stop 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 -- 172.21.15.178:0/1688842984 shutdown_connections 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 --2- 172.21.15.178:0/1688842984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f37880f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 -- 172.21.15.178:0/1688842984 >> 172.21.15.178:0/1688842984 conn(0x7f37880f0b70 msgr2=0x7f37880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 -- 172.21.15.178:0/1688842984 shutdown_connections 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.546+0000 7f378e26f700 1 -- 172.21.15.178:0/1688842984 wait complete. 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f378e26f700 1 Processor -- start 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f378e26f700 1 -- start start 2024-01-21T11:21:31.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f378e26f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f3788109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f378e26f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3788060240 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f3787fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f3788109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f3787fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f3788109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:41912/0 (socket says 172.21.15.178:41912) 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.547+0000 7f3787fff700 1 -- 172.21.15.178:0/1134965326 learned_addr learned my addr 172.21.15.178:0/1134965326 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f3787fff700 1 -- 172.21.15.178:0/1134965326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3778005040 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f3787fff700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f3788109430 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3778013510 tx=0x7f377800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3778009320 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3788060440 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3778013f00 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3778014d90 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.548+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3788109ae0 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.549+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37780263a0 con 0x7f37880f5410 2024-01-21T11:21:31.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.549+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3788045ad0 con 0x7f37880f5410 2024-01-21T11:21:31.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.549+0000 7f3784ff9700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f377002ff60 0x7f3770032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:31.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.549+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3778041100 con 0x7f37880f5410 2024-01-21T11:21:31.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.549+0000 7f37877fe700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f377002ff60 0x7f3770032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:31.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.549+0000 7f37877fe700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f377002ff60 0x7f3770032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f377c0096a0 tx=0x7f377c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:31.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.553+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37780126f0 con 0x7f37880f5410 2024-01-21T11:21:31.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.761+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37880fae60 con 0x7f37880f5410 2024-01-21T11:21:31.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.762+0000 7f3784ff9700 1 -- 172.21.15.178:0/1134965326 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3778012330 con 0x7f37880f5410 2024-01-21T11:21:31.763 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:31.763 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f377002ff60 msgr2=0x7f3770032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f377002ff60 0x7f3770032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f377c0096a0 tx=0x7f377c006b40).stop 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 msgr2=0x7f3788109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f3788109430 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3778013510 tx=0x7f377800b040).stop 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 shutdown_connections 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f377002ff60 0x7f3770032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 --2- 172.21.15.178:0/1134965326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37880f5410 0x7f3788109430 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 >> 172.21.15.178:0/1134965326 conn(0x7f37880f0b70 msgr2=0x7f37880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 shutdown_connections 2024-01-21T11:21:31.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:31.764+0000 7f378e26f700 1 -- 172.21.15.178:0/1134965326 wait complete. 2024-01-21T11:21:31.766 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:33.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:33.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:33.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:31.715578+0000 mgr.smithi089.ufybqw (mgr.14162) 255 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:33.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:32 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:31.763727+0000 mon.smithi089 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.178:0/1134965326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:34.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- 172.21.15.178:0/1181327709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 msgr2=0x7f6ff00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:34.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 --2- 172.21.15.178:0/1181327709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff00f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6fe0009fb0 tx=0x7f6fe0009b30).stop 2024-01-21T11:21:34.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- 172.21.15.178:0/1181327709 shutdown_connections 2024-01-21T11:21:34.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 --2- 172.21.15.178:0/1181327709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff00f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:34.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- 172.21.15.178:0/1181327709 >> 172.21.15.178:0/1181327709 conn(0x7f6ff00f09a0 msgr2=0x7f6ff00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:34.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- 172.21.15.178:0/1181327709 shutdown_connections 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- 172.21.15.178:0/1181327709 wait complete. 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 Processor -- start 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- start start 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.267+0000 7f6ff7155700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff0060240 con 0x7f6ff00f51a0 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff6153700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff6153700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:32782/0 (socket says 172.21.15.178:32782) 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff6153700 1 -- 172.21.15.178:0/3576966471 learned_addr learned my addr 172.21.15.178:0/3576966471 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff6153700 1 -- 172.21.15.178:0/3576966471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0005040 con 0x7f6ff00f51a0 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff6153700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff01091b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6fe0013510 tx=0x7f6fe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe0009320 con 0x7f6ff00f51a0 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fe0013f00 con 0x7f6ff00f51a0 2024-01-21T11:21:34.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff0060440 con 0x7f6ff00f51a0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe0014dc0 con 0x7f6ff00f51a0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.268+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff0109860 con 0x7f6ff00f51a0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.269+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fe00095f0 con 0x7f6ff00f51a0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.269+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff0045ad0 con 0x7f6ff00f51a0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.269+0000 7f6fe6ffd700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fdc02ff60 0x7f6fdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.269+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fe00410c0 con 0x7f6ff00f51a0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.269+0000 7f6ff5952700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fdc02ff60 0x7f6fdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:34.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.270+0000 7f6ff5952700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fdc02ff60 0x7f6fdc032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f6fec0096a0 tx=0x7f6fec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:34.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.274+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fe0014420 con 0x7f6ff00f51a0 2024-01-21T11:21:34.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.482+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ff010a080 con 0x7f6ff00f51a0 2024-01-21T11:21:34.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.483+0000 7f6fe6ffd700 1 -- 172.21.15.178:0/3576966471 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fe0010230 con 0x7f6ff00f51a0 2024-01-21T11:21:34.484 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:34.484 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fdc02ff60 msgr2=0x7f6fdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fdc02ff60 0x7f6fdc032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f6fec0096a0 tx=0x7f6fec006b40).stop 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 msgr2=0x7f6ff01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff01091b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6fe0013510 tx=0x7f6fe000b040).stop 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 shutdown_connections 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f6fdc02ff60 0x7f6fdc032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 --2- 172.21.15.178:0/3576966471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff00f51a0 0x7f6ff01091b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 >> 172.21.15.178:0/3576966471 conn(0x7f6ff00f09a0 msgr2=0x7f6ff00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 shutdown_connections 2024-01-21T11:21:34.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:34.485+0000 7f6ff7155700 1 -- 172.21.15.178:0/3576966471 wait complete. 2024-01-21T11:21:34.487 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:35.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:33.715921+0000 mgr.smithi089.ufybqw (mgr.14162) 256 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:35.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:34 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:34.484891+0000 mon.smithi089 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.178:0/3576966471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:35.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:35.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 -- 172.21.15.178:0/324648064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 msgr2=0x7f3af40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 --2- 172.21.15.178:0/324648064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af40f66c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f3ae8009fb0 tx=0x7f3ae8009b30).stop 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 -- 172.21.15.178:0/324648064 shutdown_connections 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 --2- 172.21.15.178:0/324648064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af40f66c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 -- 172.21.15.178:0/324648064 >> 172.21.15.178:0/324648064 conn(0x7f3af40f1aa0 msgr2=0x7f3af40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 -- 172.21.15.178:0/324648064 shutdown_connections 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 -- 172.21.15.178:0/324648064 wait complete. 2024-01-21T11:21:37.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.034+0000 7f3af3fff700 1 Processor -- start 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af3fff700 1 -- start start 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af3fff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af41069a0 con 0x7f3af40f62a0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:32808/0 (socket says 172.21.15.178:32808) 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af2ffd700 1 -- 172.21.15.178:0/998755644 learned_addr learned my addr 172.21.15.178:0/998755644 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af2ffd700 1 -- 172.21.15.178:0/998755644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8005040 con 0x7f3af40f62a0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.035+0000 7f3af2ffd700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af41064b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3ae8013510 tx=0x7f3ae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.036+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae8009320 con 0x7f3af40f62a0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.036+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ae8013f00 con 0x7f3af40f62a0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.036+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af4106ba0 con 0x7f3af40f62a0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.036+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae8014dc0 con 0x7f3af40f62a0 2024-01-21T11:21:37.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.036+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4107030 con 0x7f3af40f62a0 2024-01-21T11:21:37.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.036+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ae8014420 con 0x7f3af40f62a0 2024-01-21T11:21:37.039 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.037+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4045ad0 con 0x7f3af40f62a0 2024-01-21T11:21:37.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.037+0000 7f3adb7fe700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3adc02ff60 0x7f3adc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:37.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.037+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ae80410f0 con 0x7f3af40f62a0 2024-01-21T11:21:37.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.037+0000 7f3af27fc700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3adc02ff60 0x7f3adc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:37.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.037+0000 7f3af27fc700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3adc02ff60 0x7f3adc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3ae40096a0 tx=0x7f3ae4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:37.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.041+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae80095f0 con 0x7f3af40f62a0 2024-01-21T11:21:37.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:36 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:35.716367+0000 mgr.smithi089.ufybqw (mgr.14162) 257 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:37.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.249+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3af4107ee0 con 0x7f3af40f62a0 2024-01-21T11:21:37.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.249+0000 7f3adb7fe700 1 -- 172.21.15.178:0/998755644 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ae801cb10 con 0x7f3af40f62a0 2024-01-21T11:21:37.250 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:37.251 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:37.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3adc02ff60 msgr2=0x7f3adc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:37.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3adc02ff60 0x7f3adc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3ae40096a0 tx=0x7f3ae4006b40).stop 2024-01-21T11:21:37.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 msgr2=0x7f3af41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:37.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af41064b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3ae8013510 tx=0x7f3ae800b040).stop 2024-01-21T11:21:37.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 shutdown_connections 2024-01-21T11:21:37.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f3adc02ff60 0x7f3adc032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:37.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 --2- 172.21.15.178:0/998755644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3af40f62a0 0x7f3af41064b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:37.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 >> 172.21.15.178:0/998755644 conn(0x7f3af40f1aa0 msgr2=0x7f3af40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:37.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 shutdown_connections 2024-01-21T11:21:37.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:37.251+0000 7f3af3fff700 1 -- 172.21.15.178:0/998755644 wait complete. 2024-01-21T11:21:37.253 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:38.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:37 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:37.251113+0000 mon.smithi089 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.178:0/998755644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:38.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:38.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:39.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:38 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:37.716834+0000 mgr.smithi089.ufybqw (mgr.14162) 258 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:39.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/2834322219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 msgr2=0x7fb1a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:39.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 --2- 172.21.15.178:0/2834322219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a00f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fb190009fb0 tx=0x7fb190009b30).stop 2024-01-21T11:21:39.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/2834322219 shutdown_connections 2024-01-21T11:21:39.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 --2- 172.21.15.178:0/2834322219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a00f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:39.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/2834322219 >> 172.21.15.178:0/2834322219 conn(0x7fb1a00f0b70 msgr2=0x7fb1a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:39.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/2834322219 shutdown_connections 2024-01-21T11:21:39.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.758+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/2834322219 wait complete. 2024-01-21T11:21:39.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.759+0000 7fb1a5bf7700 1 Processor -- start 2024-01-21T11:21:39.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.759+0000 7fb1a5bf7700 1 -- start start 2024-01-21T11:21:39.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.759+0000 7fb1a5bf7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a01093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:39.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.759+0000 7fb1a5bf7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a0060280 con 0x7fb1a00f5410 2024-01-21T11:21:39.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.759+0000 7fb19f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a01093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:39.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb19f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a01093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:32828/0 (socket says 172.21.15.178:32828) 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb19f7fe700 1 -- 172.21.15.178:0/864043007 learned_addr learned my addr 172.21.15.178:0/864043007 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb19f7fe700 1 -- 172.21.15.178:0/864043007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190005040 con 0x7fb1a00f5410 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb19f7fe700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a01093c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb190013510 tx=0x7fb19000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb190009320 con 0x7fb1a00f5410 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a0060480 con 0x7fb1a00f5410 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb190013f00 con 0x7fb1a00f5410 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb190014d90 con 0x7fb1a00f5410 2024-01-21T11:21:39.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.760+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a0109af0 con 0x7fb1a00f5410 2024-01-21T11:21:39.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.761+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1900095f0 con 0x7fb1a00f5410 2024-01-21T11:21:39.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.761+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a0045ad0 con 0x7fb1a00f5410 2024-01-21T11:21:39.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.761+0000 7fb197fff700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb18c02ffb0 0x7fb18c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:39.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.761+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1900410c0 con 0x7fb1a00f5410 2024-01-21T11:21:39.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.761+0000 7fb19effd700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb18c02ffb0 0x7fb18c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:39.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.762+0000 7fb19effd700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb18c02ffb0 0x7fb18c032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb1880096a0 tx=0x7fb188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:39.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.766+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb190014420 con 0x7fb1a00f5410 2024-01-21T11:21:39.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.973+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1a010a310 con 0x7fb1a00f5410 2024-01-21T11:21:39.975 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.974+0000 7fb197fff700 1 -- 172.21.15.178:0/864043007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb19001d7f0 con 0x7fb1a00f5410 2024-01-21T11:21:39.975 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:39.975 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb18c02ffb0 msgr2=0x7fb18c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb18c02ffb0 0x7fb18c032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb1880096a0 tx=0x7fb188006b40).stop 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 msgr2=0x7fb1a01093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a01093c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb190013510 tx=0x7fb19000b040).stop 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 shutdown_connections 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb18c02ffb0 0x7fb18c032470 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 --2- 172.21.15.178:0/864043007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a00f5410 0x7fb1a01093c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 >> 172.21.15.178:0/864043007 conn(0x7fb1a00f0b70 msgr2=0x7fb1a00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:39.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 shutdown_connections 2024-01-21T11:21:39.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:39.976+0000 7fb1a5bf7700 1 -- 172.21.15.178:0/864043007 wait complete. 2024-01-21T11:21:39.978 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:40.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:21:38.223632+0000 mgr.smithi089.ufybqw (mgr.14162) 259 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi089 ... 2024-01-21T11:21:40.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Verifying port 9100 ... 2024-01-21T11:21:40.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Non-zero exit code 1 from systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089 2024-01-21T11:21:40.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: systemctl: stderr Job for ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089.service failed because a timeout was exceeded. 2024-01-21T11:21:40.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: systemctl: stderr See "systemctl status ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089.service" and "journalctl -xe" for details. 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Traceback (most recent call last): 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: main() 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: r = ctx.func(ctx) 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: return func(ctx) 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ports=daemon_ports) 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T11:21:40.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: RuntimeError: Failed command: systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Traceback (most recent call last): 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: yield (conn, connr) 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: code, '\n'.join(err))) 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi089 ... 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Verifying port 9100 ... 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Non-zero exit code 1 from systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: systemctl: stderr Job for ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089.service failed because a timeout was exceeded. 2024-01-21T11:21:40.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: systemctl: stderr See "systemctl status ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089.service" and "journalctl -xe" for details. 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: Traceback (most recent call last): 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: main() 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: r = ctx.func(ctx) 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: return func(ctx) 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: ports=daemon_ports) 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T11:21:40.187 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: File "/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T11:21:40.188 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T11:21:40.188 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: RuntimeError: Failed command: systemctl start ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@node-exporter.smithi089 2024-01-21T11:21:40.188 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:39 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cephadm 2024-01-21T11:21:38.229152+0000 mgr.smithi089.ufybqw (mgr.14162) 260 : cephadm [INF] Deploying daemon prometheus.smithi089 on smithi089 2024-01-21T11:21:41.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:39.717249+0000 mgr.smithi089.ufybqw (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:41.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:40 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:39.975860+0000 mon.smithi089 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.178:0/864043007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:41.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:41.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:42.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 -- 172.21.15.178:0/1884841605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 msgr2=0x7f8ae80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:42.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 --2- 172.21.15.178:0/1884841605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae80f90f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f8ad8009fb0 tx=0x7f8ad8009b30).stop 2024-01-21T11:21:42.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 -- 172.21.15.178:0/1884841605 shutdown_connections 2024-01-21T11:21:42.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 --2- 172.21.15.178:0/1884841605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae80f90f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:42.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 -- 172.21.15.178:0/1884841605 >> 172.21.15.178:0/1884841605 conn(0x7f8ae80f0b70 msgr2=0x7f8ae80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 -- 172.21.15.178:0/1884841605 shutdown_connections 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.486+0000 7f8aede81700 1 -- 172.21.15.178:0/1884841605 wait complete. 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8aede81700 1 Processor -- start 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8aede81700 1 -- start start 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8aede81700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8aede81700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8106bd0 con 0x7f8ae8060280 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8ae77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8ae77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57124/0 (socket says 172.21.15.178:57124) 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.487+0000 7f8ae77fe700 1 -- 172.21.15.178:0/1452934462 learned_addr learned my addr 172.21.15.178:0/1452934462 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8ae77fe700 1 -- 172.21.15.178:0/1452934462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad8005040 con 0x7f8ae8060280 2024-01-21T11:21:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8ae77fe700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae81066e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f8ad8013510 tx=0x7f8ad800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ad8009320 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae8106dd0 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ad8013f00 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ad8014dc0 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.488+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae8107260 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.489+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ad80095f0 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.489+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae8045ad0 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.489+0000 7f8acffff700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8ad002ff60 0x7f8ad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.489+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ad8041000 con 0x7f8ae8060280 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.489+0000 7f8ae6ffd700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8ad002ff60 0x7f8ad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.489+0000 7f8ae6ffd700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8ad002ff60 0x7f8ad0032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8adc0096a0 tx=0x7f8adc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:42.494 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.493+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ad801c790 con 0x7f8ae8060280 2024-01-21T11:21:42.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.701+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ae8108110 con 0x7f8ae8060280 2024-01-21T11:21:42.703 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.702+0000 7f8acffff700 1 -- 172.21.15.178:0/1452934462 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ad8010230 con 0x7f8ae8060280 2024-01-21T11:21:42.703 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:42.703 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.703+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8ad002ff60 msgr2=0x7f8ad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8ad002ff60 0x7f8ad0032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8adc0096a0 tx=0x7f8adc006b40).stop 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 msgr2=0x7f8ae81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae81066e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f8ad8013510 tx=0x7f8ad800b040).stop 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 shutdown_connections 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f8ad002ff60 0x7f8ad0032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 --2- 172.21.15.178:0/1452934462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ae8060280 0x7f8ae81066e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 >> 172.21.15.178:0/1452934462 conn(0x7f8ae80f0b70 msgr2=0x7f8ae80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 shutdown_connections 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:42.704+0000 7f8aede81700 1 -- 172.21.15.178:0/1452934462 wait complete. 2024-01-21T11:21:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:43.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:41.717626+0000 mgr.smithi089.ufybqw (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:43.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:42 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:42.703734+0000 mon.smithi089 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.178:0/1452934462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:44.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:44.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:45.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:44 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:43.717980+0000 mgr.smithi089.ufybqw (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 -- 172.21.15.178:0/2444927077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 msgr2=0x7f89100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 --2- 172.21.15.178:0/2444927077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f890c009fb0 tx=0x7f890c009b30).stop 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 -- 172.21.15.178:0/2444927077 shutdown_connections 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 --2- 172.21.15.178:0/2444927077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 -- 172.21.15.178:0/2444927077 >> 172.21.15.178:0/2444927077 conn(0x7f89100f0b70 msgr2=0x7f89100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 -- 172.21.15.178:0/2444927077 shutdown_connections 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.273+0000 7f8917f53700 1 -- 172.21.15.178:0/2444927077 wait complete. 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8917f53700 1 Processor -- start 2024-01-21T11:21:45.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8917f53700 1 -- start start 2024-01-21T11:21:45.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8917f53700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:45.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8917f53700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89100fb820 con 0x7f89100f5410 2024-01-21T11:21:45.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8915cef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:45.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8915cef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57152/0 (socket says 172.21.15.178:57152) 2024-01-21T11:21:45.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.274+0000 7f8915cef700 1 -- 172.21.15.178:0/962689579 learned_addr learned my addr 172.21.15.178:0/962689579 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:45.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f8915cef700 1 -- 172.21.15.178:0/962689579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c005040 con 0x7f89100f5410 2024-01-21T11:21:45.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f8915cef700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100fd3a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f890c014000 tx=0x7f890c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f890c0095f0 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89100fb9c0 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f890c007e70 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f890c014ea0 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.275+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89100fbe50 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.276+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f890c025070 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.276+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8910059100 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.276+0000 7f89067fc700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f88fc02ff60 0x7f88fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.276+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f890c040a10 con 0x7f89100f5410 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.276+0000 7f89154ee700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f88fc02ff60 0x7f88fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:45.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.276+0000 7f89154ee700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f88fc02ff60 0x7f88fc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f89000096a0 tx=0x7f8900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:45.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.281+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f890c00f4e0 con 0x7f89100f5410 2024-01-21T11:21:45.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.489+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8910045ad0 con 0x7f89100f5410 2024-01-21T11:21:45.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.489+0000 7f89067fc700 1 -- 172.21.15.178:0/962689579 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f890c00f300 con 0x7f89100f5410 2024-01-21T11:21:45.490 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:45.491 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f88fc02ff60 msgr2=0x7f88fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f88fc02ff60 0x7f88fc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f89000096a0 tx=0x7f8900006b40).stop 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 msgr2=0x7f89100fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100fd3a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f890c014000 tx=0x7f890c00b040).stop 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 shutdown_connections 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f88fc02ff60 0x7f88fc032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 --2- 172.21.15.178:0/962689579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89100f5410 0x7f89100fd3a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 >> 172.21.15.178:0/962689579 conn(0x7f89100f0b70 msgr2=0x7f891005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:45.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 shutdown_connections 2024-01-21T11:21:45.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:45.491+0000 7f8917f53700 1 -- 172.21.15.178:0/962689579 wait complete. 2024-01-21T11:21:45.493 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:46.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:45 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:45.491240+0000 mon.smithi089 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.178:0/962689579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:46.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:46.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:47.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:46 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:45.718342+0000 mgr.smithi089.ufybqw (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 -- 172.21.15.178:0/2942577548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 msgr2=0x7f9c2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 --2- 172.21.15.178:0/2942577548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c0f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f9c20009fb0 tx=0x7f9c20009b30).stop 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 -- 172.21.15.178:0/2942577548 shutdown_connections 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 --2- 172.21.15.178:0/2942577548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c0f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 -- 172.21.15.178:0/2942577548 >> 172.21.15.178:0/2942577548 conn(0x7f9c2c0f0b70 msgr2=0x7f9c2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 -- 172.21.15.178:0/2942577548 shutdown_connections 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.019+0000 7f9c32edc700 1 -- 172.21.15.178:0/2942577548 wait complete. 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.020+0000 7f9c32edc700 1 Processor -- start 2024-01-21T11:21:48.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.020+0000 7f9c32edc700 1 -- start start 2024-01-21T11:21:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.020+0000 7f9c32edc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.020+0000 7f9c32edc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c106c10 con 0x7f9c2c0f5410 2024-01-21T11:21:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c30c78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c30c78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57166/0 (socket says 172.21.15.178:57166) 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c30c78700 1 -- 172.21.15.178:0/1946331048 learned_addr learned my addr 172.21.15.178:0/1946331048 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c30c78700 1 -- 172.21.15.178:0/1946331048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c20005040 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c30c78700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c106720 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f9c20013510 tx=0x7f9c2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c20009320 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c106e10 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c20013f00 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c20014d90 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.021+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c1072a0 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.022+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c200095f0 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.022+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c045ad0 con 0x7f9c2c0f5410 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.022+0000 7f9c297fa700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c1c02ffb0 0x7f9c1c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:48.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.022+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c200410a0 con 0x7f9c2c0f5410 2024-01-21T11:21:48.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.023+0000 7f9c2bfff700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c1c02ffb0 0x7f9c1c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:48.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.023+0000 7f9c2bfff700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c1c02ffb0 0x7f9c1c032470 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9c180096a0 tx=0x7f9c18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:48.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.027+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c2006d2b0 con 0x7f9c2c0f5410 2024-01-21T11:21:48.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:47 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:47.718807+0000 mgr.smithi089.ufybqw (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:48.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.235+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c2c108150 con 0x7f9c2c0f5410 2024-01-21T11:21:48.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.236+0000 7f9c297fa700 1 -- 172.21.15.178:0/1946331048 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c2001d7f0 con 0x7f9c2c0f5410 2024-01-21T11:21:48.237 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:48.237 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:48.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c1c02ffb0 msgr2=0x7f9c1c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:48.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c1c02ffb0 0x7f9c1c032470 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9c180096a0 tx=0x7f9c18006b40).stop 2024-01-21T11:21:48.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 msgr2=0x7f9c2c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:48.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c106720 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f9c20013510 tx=0x7f9c2000b040).stop 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 shutdown_connections 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f9c1c02ffb0 0x7f9c1c032470 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 --2- 172.21.15.178:0/1946331048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c2c0f5410 0x7f9c2c106720 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.237+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 >> 172.21.15.178:0/1946331048 conn(0x7f9c2c0f0b70 msgr2=0x7f9c2c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.238+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 shutdown_connections 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:48.238+0000 7f9c32edc700 1 -- 172.21.15.178:0/1946331048 wait complete. 2024-01-21T11:21:48.239 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:49.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:48 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:48.237627+0000 mon.smithi089 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.178:0/1946331048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:49.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:49.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:50.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 -- 172.21.15.178:0/742119246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 msgr2=0x7f584c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 --2- 172.21.15.178:0/742119246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c0f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f5840009fb0 tx=0x7f5840009b30).stop 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 -- 172.21.15.178:0/742119246 shutdown_connections 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 --2- 172.21.15.178:0/742119246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c0f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 -- 172.21.15.178:0/742119246 >> 172.21.15.178:0/742119246 conn(0x7f584c0f1c70 msgr2=0x7f584c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 -- 172.21.15.178:0/742119246 shutdown_connections 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.784+0000 7f5852d16700 1 -- 172.21.15.178:0/742119246 wait complete. 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5852d16700 1 Processor -- start 2024-01-21T11:21:50.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5852d16700 1 -- start start 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5852d16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5852d16700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c069e20 con 0x7f584c0f6510 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5850ab2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5850ab2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:57180/0 (socket says 172.21.15.178:57180) 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.785+0000 7f5850ab2700 1 -- 172.21.15.178:0/3989994193 learned_addr learned my addr 172.21.15.178:0/3989994193 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f5850ab2700 1 -- 172.21.15.178:0/3989994193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840005040 con 0x7f584c0f6510 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f5850ab2700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c06b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f5840013510 tx=0x7f584000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5840009320 con 0x7f584c0f6510 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584c069fc0 con 0x7f584c0f6510 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5840013f00 con 0x7f584c0f6510 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5840014d90 con 0x7f584c0f6510 2024-01-21T11:21:50.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.786+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584c06a450 con 0x7f584c0f6510 2024-01-21T11:21:50.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.787+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5840014420 con 0x7f584c0f6510 2024-01-21T11:21:50.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.787+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584c059100 con 0x7f584c0f6510 2024-01-21T11:21:50.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.787+0000 7f58497fa700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f583c02ff60 0x7f583c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:50.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.787+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58400410c0 con 0x7f584c0f6510 2024-01-21T11:21:50.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.787+0000 7f584bfff700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f583c02ff60 0x7f583c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:50.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.788+0000 7f584bfff700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f583c02ff60 0x7f583c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f58380096a0 tx=0x7f5838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:50.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.792+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5840026bf0 con 0x7f584c0f6510 2024-01-21T11:21:50.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.996+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f584c0fbf60 con 0x7f584c0f6510 2024-01-21T11:21:50.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.997+0000 7f58497fa700 1 -- 172.21.15.178:0/3989994193 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f584001db70 con 0x7f584c0f6510 2024-01-21T11:21:50.998 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:50.998 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:50.999 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f583c02ff60 msgr2=0x7f583c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:50.999 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f583c02ff60 0x7f583c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f58380096a0 tx=0x7f5838006b40).stop 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 msgr2=0x7f584c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c06b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f5840013510 tx=0x7f584000b040).stop 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 shutdown_connections 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f583c02ff60 0x7f583c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 --2- 172.21.15.178:0/3989994193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f584c0f6510 0x7f584c06b9a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 >> 172.21.15.178:0/3989994193 conn(0x7f584c0f1c70 msgr2=0x7f584c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 shutdown_connections 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:50.999+0000 7f5852d16700 1 -- 172.21.15.178:0/3989994193 wait complete. 2024-01-21T11:21:51.000 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:51.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:50 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:49.719260+0000 mgr.smithi089.ufybqw (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:52.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:51 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:50.998888+0000 mon.smithi089 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.178:0/3989994193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:52.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:52.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:53.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:52 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:51.719675+0000 mgr.smithi089.ufybqw (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 -- 172.21.15.178:0/1750779573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 msgr2=0x7fab880fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 --2- 172.21.15.178:0/1750779573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab880fb550 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fab78009fb0 tx=0x7fab78009b30).stop 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 -- 172.21.15.178:0/1750779573 shutdown_connections 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 --2- 172.21.15.178:0/1750779573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab880fb550 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 -- 172.21.15.178:0/1750779573 >> 172.21.15.178:0/1750779573 conn(0x7fab880f0b70 msgr2=0x7fab880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 -- 172.21.15.178:0/1750779573 shutdown_connections 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.543+0000 7fab901de700 1 -- 172.21.15.178:0/1750779573 wait complete. 2024-01-21T11:21:53.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab901de700 1 Processor -- start 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab901de700 1 -- start start 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab901de700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab881088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab901de700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab88108db0 con 0x7fab88060220 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab8df7a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab881088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab8df7a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab881088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49072/0 (socket says 172.21.15.178:49072) 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab8df7a700 1 -- 172.21.15.178:0/3170473755 learned_addr learned my addr 172.21.15.178:0/3170473755 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.544+0000 7fab8df7a700 1 -- 172.21.15.178:0/3170473755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab78005040 con 0x7fab88060220 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.545+0000 7fab8df7a700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab881088c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fab78013510 tx=0x7fab7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.545+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab78009320 con 0x7fab88060220 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.545+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab88108fb0 con 0x7fab88060220 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.545+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab78013f00 con 0x7fab88060220 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.545+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab78014dc0 con 0x7fab88060220 2024-01-21T11:21:53.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.545+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab88109440 con 0x7fab88060220 2024-01-21T11:21:53.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.546+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab780095f0 con 0x7fab88060220 2024-01-21T11:21:53.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.546+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab88045ad0 con 0x7fab88060220 2024-01-21T11:21:53.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.546+0000 7fab7e7fc700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fab7402ff60 0x7fab74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:53.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.546+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab78041040 con 0x7fab88060220 2024-01-21T11:21:53.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.546+0000 7fab8d779700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fab7402ff60 0x7fab74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:53.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.546+0000 7fab8d779700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fab7402ff60 0x7fab74032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fab840096a0 tx=0x7fab84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:53.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.551+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab78014420 con 0x7fab88060220 2024-01-21T11:21:53.759 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.760+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab8810a2f0 con 0x7fab88060220 2024-01-21T11:21:53.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.761+0000 7fab7e7fc700 1 -- 172.21.15.178:0/3170473755 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab78010230 con 0x7fab88060220 2024-01-21T11:21:53.760 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:53.761 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fab7402ff60 msgr2=0x7fab74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fab7402ff60 0x7fab74032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fab840096a0 tx=0x7fab84006b40).stop 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 msgr2=0x7fab881088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab881088c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fab78013510 tx=0x7fab7800b040).stop 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 shutdown_connections 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fab7402ff60 0x7fab74032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 --2- 172.21.15.178:0/3170473755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab88060220 0x7fab881088c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 >> 172.21.15.178:0/3170473755 conn(0x7fab880f0b70 msgr2=0x7fab880f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 shutdown_connections 2024-01-21T11:21:53.762 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:53.762+0000 7fab901de700 1 -- 172.21.15.178:0/3170473755 wait complete. 2024-01-21T11:21:53.763 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:55.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:55.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:55.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:53.719980+0000 mgr.smithi089.ufybqw (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:55.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:54 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:53.761388+0000 mon.smithi089 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.178:0/3170473755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:56.313 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.313+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1618842456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 msgr2=0x7fb4f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:56.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 --2- 172.21.15.178:0/1618842456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f80f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb4ec009fb0 tx=0x7fb4ec009b30).stop 2024-01-21T11:21:56.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1618842456 shutdown_connections 2024-01-21T11:21:56.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 --2- 172.21.15.178:0/1618842456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f80f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:56.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1618842456 >> 172.21.15.178:0/1618842456 conn(0x7fb4f80f0b70 msgr2=0x7fb4f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:56.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1618842456 shutdown_connections 2024-01-21T11:21:56.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1618842456 wait complete. 2024-01-21T11:21:56.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 Processor -- start 2024-01-21T11:21:56.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.314+0000 7fb4fed4a700 1 -- start start 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fed4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fed4a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8105b20 con 0x7fb4f80f5410 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fcae6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fcae6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49088/0 (socket says 172.21.15.178:49088) 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fcae6700 1 -- 172.21.15.178:0/1915657814 learned_addr learned my addr 172.21.15.178:0/1915657814 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fcae6700 1 -- 172.21.15.178:0/1915657814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec005040 con 0x7fb4f80f5410 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.315+0000 7fb4fcae6700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f8105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb4ec013510 tx=0x7fb4ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.316+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ec009320 con 0x7fb4f80f5410 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.316+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f8105d20 con 0x7fb4f80f5410 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.316+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4ec013f00 con 0x7fb4f80f5410 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.316+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ec014d90 con 0x7fb4f80f5410 2024-01-21T11:21:56.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.316+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f81061b0 con 0x7fb4f80f5410 2024-01-21T11:21:56.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.316+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4ec0263a0 con 0x7fb4f80f5410 2024-01-21T11:21:56.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.317+0000 7fb4f57fa700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4e802ff60 0x7fb4e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:56.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.317+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4ec0410d0 con 0x7fb4f80f5410 2024-01-21T11:21:56.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.317+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f8045ad0 con 0x7fb4f80f5410 2024-01-21T11:21:56.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.317+0000 7fb4f7fff700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4e802ff60 0x7fb4e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:56.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.317+0000 7fb4f7fff700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4e802ff60 0x7fb4e8032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb4e40096a0 tx=0x7fb4e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:56.321 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.322+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4ec012920 con 0x7fb4f80f5410 2024-01-21T11:21:56.530 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.530+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4f80fae60 con 0x7fb4f80f5410 2024-01-21T11:21:56.530 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.530+0000 7fb4f57fa700 1 -- 172.21.15.178:0/1915657814 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4ec02b020 con 0x7fb4f80f5410 2024-01-21T11:21:56.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:56.531 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4e802ff60 msgr2=0x7fb4e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4e802ff60 0x7fb4e8032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb4e40096a0 tx=0x7fb4e4006b40).stop 2024-01-21T11:21:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 msgr2=0x7fb4f8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f8105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb4ec013510 tx=0x7fb4ec00b040).stop 2024-01-21T11:21:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 shutdown_connections 2024-01-21T11:21:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb4e802ff60 0x7fb4e8032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:56.533 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 --2- 172.21.15.178:0/1915657814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f80f5410 0x7fb4f8105630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:56.533 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 >> 172.21.15.178:0/1915657814 conn(0x7fb4f80f0b70 msgr2=0x7fb4f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:56.533 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 shutdown_connections 2024-01-21T11:21:56.533 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:56.532+0000 7fb4fed4a700 1 -- 172.21.15.178:0/1915657814 wait complete. 2024-01-21T11:21:56.533 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:21:57.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:55.720368+0000 mgr.smithi089.ufybqw (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:57.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:56 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:56.531240+0000 mon.smithi089 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.178:0/1915657814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:21:57.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:21:57.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 -- 172.21.15.178:0/669388079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 msgr2=0x7fa6a80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 --2- 172.21.15.178:0/669388079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a80f9190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fa698009fb0 tx=0x7fa698009b30).stop 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 -- 172.21.15.178:0/669388079 shutdown_connections 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 --2- 172.21.15.178:0/669388079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a80f9190 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 -- 172.21.15.178:0/669388079 >> 172.21.15.178:0/669388079 conn(0x7fa6a80f0b70 msgr2=0x7fa6a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 -- 172.21.15.178:0/669388079 shutdown_connections 2024-01-21T11:21:59.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.116+0000 7fa6adfad700 1 -- 172.21.15.178:0/669388079 wait complete. 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.117+0000 7fa6adfad700 1 Processor -- start 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.117+0000 7fa6adfad700 1 -- start start 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.117+0000 7fa6adfad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a81088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.117+0000 7fa6adfad700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8108dc0 con 0x7fa6a80601e0 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6a77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a81088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6a77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a81088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49108/0 (socket says 172.21.15.178:49108) 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6a77fe700 1 -- 172.21.15.178:0/15391398 learned_addr learned my addr 172.21.15.178:0/15391398 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6a77fe700 1 -- 172.21.15.178:0/15391398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698005040 con 0x7fa6a80601e0 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6a77fe700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a81088d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fa698004dd0 tx=0x7fa698009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa698018070 con 0x7fa6a80601e0 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a8108fc0 con 0x7fa6a80601e0 2024-01-21T11:21:59.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa698013410 con 0x7fa6a80601e0 2024-01-21T11:21:59.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa69801c3c0 con 0x7fa6a80601e0 2024-01-21T11:21:59.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.118+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a8109450 con 0x7fa6a80601e0 2024-01-21T11:21:59.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.119+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa69801c630 con 0x7fa6a80601e0 2024-01-21T11:21:59.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.119+0000 7fa68ffff700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa69002ff60 0x7fa690032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:21:59.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.119+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6980413d0 con 0x7fa6a80601e0 2024-01-21T11:21:59.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.119+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a8045ad0 con 0x7fa6a80601e0 2024-01-21T11:21:59.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.119+0000 7fa6a6ffd700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa69002ff60 0x7fa690032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:21:59.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.120+0000 7fa6a6ffd700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa69002ff60 0x7fa690032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa69c0096a0 tx=0x7fa69c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:21:59.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.124+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa69801ca30 con 0x7fa6a80601e0 2024-01-21T11:21:59.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:58 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:57.720789+0000 mgr.smithi089.ufybqw (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:21:59.328 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.328+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6a810a190 con 0x7fa6a80601e0 2024-01-21T11:21:59.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.329+0000 7fa68ffff700 1 -- 172.21.15.178:0/15391398 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa69801b9c0 con 0x7fa6a80601e0 2024-01-21T11:21:59.329 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:21:59.330 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa69002ff60 msgr2=0x7fa690032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa69002ff60 0x7fa690032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa69c0096a0 tx=0x7fa69c006b40).stop 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 msgr2=0x7fa6a81088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a81088d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fa698004dd0 tx=0x7fa698009b30).stop 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 shutdown_connections 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fa69002ff60 0x7fa690032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 --2- 172.21.15.178:0/15391398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a80601e0 0x7fa6a81088d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 >> 172.21.15.178:0/15391398 conn(0x7fa6a80f0b70 msgr2=0x7fa6a80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 shutdown_connections 2024-01-21T11:21:59.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:21:59.331+0000 7fa6adfad700 1 -- 172.21.15.178:0/15391398 wait complete. 2024-01-21T11:21:59.332 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:22:00.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:21:59 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:21:59.329948+0000 mon.smithi089 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.178:0/15391398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:22:00.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:22:00.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:22:01.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:00 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:21:59.721132+0000 mgr.smithi089.ufybqw (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 -- 172.21.15.178:0/1060763149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 msgr2=0x7fe0200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 --2- 172.21.15.178:0/1060763149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe0200f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe01c009fb0 tx=0x7fe01c009b30).stop 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 -- 172.21.15.178:0/1060763149 shutdown_connections 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 --2- 172.21.15.178:0/1060763149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe0200f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 -- 172.21.15.178:0/1060763149 >> 172.21.15.178:0/1060763149 conn(0x7fe0200f0b70 msgr2=0x7fe0200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 -- 172.21.15.178:0/1060763149 shutdown_connections 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.833+0000 7fe027577700 1 -- 172.21.15.178:0/1060763149 wait complete. 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe027577700 1 Processor -- start 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe027577700 1 -- start start 2024-01-21T11:22:01.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe027577700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe020106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe027577700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe020106c50 con 0x7fe0200f5410 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe025313700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe020106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe025313700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe020106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:49120/0 (socket says 172.21.15.178:49120) 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe025313700 1 -- 172.21.15.178:0/3615344104 learned_addr learned my addr 172.21.15.178:0/3615344104 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe025313700 1 -- 172.21.15.178:0/3615344104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c005040 con 0x7fe0200f5410 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.834+0000 7fe025313700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe020106760 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe01c009f80 tx=0x7fe01c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.835+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe01c009320 con 0x7fe0200f5410 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.835+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe01c013ec0 con 0x7fe0200f5410 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.835+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe020106e50 con 0x7fe0200f5410 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.835+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe01c01ece0 con 0x7fe0200f5410 2024-01-21T11:22:01.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.835+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0201072e0 con 0x7fe0200f5410 2024-01-21T11:22:01.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.836+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe01c01e420 con 0x7fe0200f5410 2024-01-21T11:22:01.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.836+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020045ad0 con 0x7fe0200f5410 2024-01-21T11:22:01.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.836+0000 7fe015ffb700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe00c02ff60 0x7fe00c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:01.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.836+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe01c040e70 con 0x7fe0200f5410 2024-01-21T11:22:01.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.836+0000 7fe024b12700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe00c02ff60 0x7fe00c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:01.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.836+0000 7fe024b12700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe00c02ff60 0x7fe00c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe0100096a0 tx=0x7fe010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:01.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:01.840+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe01c01e730 con 0x7fe0200f5410 2024-01-21T11:22:02.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.045+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe020108190 con 0x7fe0200f5410 2024-01-21T11:22:02.046 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.046+0000 7fe015ffb700 1 -- 172.21.15.178:0/3615344104 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe01c004490 con 0x7fe0200f5410 2024-01-21T11:22:02.046 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:22:02.046 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:22:02.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe00c02ff60 msgr2=0x7fe00c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:02.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe00c02ff60 0x7fe00c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe0100096a0 tx=0x7fe010006b40).stop 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 msgr2=0x7fe020106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe020106760 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe01c009f80 tx=0x7fe01c00b040).stop 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 shutdown_connections 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fe00c02ff60 0x7fe00c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 --2- 172.21.15.178:0/3615344104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0200f5410 0x7fe020106760 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 >> 172.21.15.178:0/3615344104 conn(0x7fe0200f0b70 msgr2=0x7fe02005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 shutdown_connections 2024-01-21T11:22:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:02.048+0000 7fe027577700 1 -- 172.21.15.178:0/3615344104 wait complete. 2024-01-21T11:22:02.049 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:22:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:22:01.721416+0000 mgr.smithi089.ufybqw (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:03.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:02 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:22:02.046966+0000 mon.smithi089 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.178:0/3615344104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:22:03.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:22:03.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 -- 172.21.15.178:0/4147350486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 msgr2=0x7f5e8c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 --2- 172.21.15.178:0/4147350486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c0f90f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f5e7c009fb0 tx=0x7f5e7c009b30).stop 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 -- 172.21.15.178:0/4147350486 shutdown_connections 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 --2- 172.21.15.178:0/4147350486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c0f90f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 -- 172.21.15.178:0/4147350486 >> 172.21.15.178:0/4147350486 conn(0x7f5e8c0f0b70 msgr2=0x7f5e8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 -- 172.21.15.178:0/4147350486 shutdown_connections 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.570+0000 7f5e94160700 1 -- 172.21.15.178:0/4147350486 wait complete. 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e94160700 1 Processor -- start 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e94160700 1 -- start start 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e94160700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:04.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e94160700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c106bd0 con 0x7f5e8c060280 2024-01-21T11:22:04.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e91efc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:04.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e91efc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56410/0 (socket says 172.21.15.178:56410) 2024-01-21T11:22:04.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.571+0000 7f5e91efc700 1 -- 172.21.15.178:0/4215385151 learned_addr learned my addr 172.21.15.178:0/4215385151 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:22:04.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e91efc700 1 -- 172.21.15.178:0/4215385151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c005040 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e91efc700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c1066e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f5e7c013510 tx=0x7f5e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c009320 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c106dd0 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e7c013f00 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c014dc0 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.572+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c107260 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.573+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e7c01d3d0 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.573+0000 7f5e827fc700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7802ffb0 0x7f5e78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.573+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e7c0410f0 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.573+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c045ad0 con 0x7f5e8c060280 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.573+0000 7f5e916fb700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7802ffb0 0x7f5e78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:04.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.574+0000 7f5e916fb700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7802ffb0 0x7f5e78032470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5e880096a0 tx=0x7f5e88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:04.577 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.578+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e7c012920 con 0x7f5e8c060280 2024-01-21T11:22:04.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.786+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e8c108110 con 0x7f5e8c060280 2024-01-21T11:22:04.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.787+0000 7f5e827fc700 1 -- 172.21.15.178:0/4215385151 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e7c02b020 con 0x7f5e8c060280 2024-01-21T11:22:04.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:22:04.787 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:22:04.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7802ffb0 msgr2=0x7f5e78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:04.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7802ffb0 0x7f5e78032470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5e880096a0 tx=0x7f5e88006b40).stop 2024-01-21T11:22:04.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 msgr2=0x7f5e8c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:04.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c1066e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f5e7c013510 tx=0x7f5e7c00b040).stop 2024-01-21T11:22:04.788 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 shutdown_connections 2024-01-21T11:22:04.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f5e7802ffb0 0x7f5e78032470 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:04.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 --2- 172.21.15.178:0/4215385151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e8c060280 0x7f5e8c1066e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:04.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 >> 172.21.15.178:0/4215385151 conn(0x7f5e8c0f0b70 msgr2=0x7f5e8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:04.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 shutdown_connections 2024-01-21T11:22:04.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:04.788+0000 7f5e94160700 1 -- 172.21.15.178:0/4215385151 wait complete. 2024-01-21T11:22:04.789 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:22:05.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:04 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:22:03.721711+0000 mgr.smithi089.ufybqw (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:06.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:22:06.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:22:06.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:05 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:22:04.787384+0000 mon.smithi089 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.178:0/4215385151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:22:07.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:06 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:22:05.722129+0000 mgr.smithi089.ufybqw (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 -- 172.21.15.178:0/1011257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 msgr2=0x7f52c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 --2- 172.21.15.178:0/1011257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c40f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f52b4009fb0 tx=0x7f52b4009b30).stop 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 -- 172.21.15.178:0/1011257626 shutdown_connections 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 --2- 172.21.15.178:0/1011257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c40f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 -- 172.21.15.178:0/1011257626 >> 172.21.15.178:0/1011257626 conn(0x7f52c40f0b70 msgr2=0x7f52c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 -- 172.21.15.178:0/1011257626 shutdown_connections 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.355+0000 7f52cbc3d700 1 -- 172.21.15.178:0/1011257626 wait complete. 2024-01-21T11:22:07.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52cbc3d700 1 Processor -- start 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52cbc3d700 1 -- start start 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52cbc3d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52cbc3d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c4105b20 con 0x7f52c40f5410 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52c99d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52c99d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56436/0 (socket says 172.21.15.178:56436) 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52c99d9700 1 -- 172.21.15.178:0/3875571841 learned_addr learned my addr 172.21.15.178:0/3875571841 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.356+0000 7f52c99d9700 1 -- 172.21.15.178:0/3875571841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4005040 con 0x7f52c40f5410 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.357+0000 7f52c99d9700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c4105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f52b4013510 tx=0x7f52b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.357+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52b4009320 con 0x7f52c40f5410 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.357+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c4105d20 con 0x7f52c40f5410 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.357+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52b4013f00 con 0x7f52c40f5410 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.357+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52b4014d90 con 0x7f52c40f5410 2024-01-21T11:22:07.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.357+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c41061b0 con 0x7f52c40f5410 2024-01-21T11:22:07.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.358+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52b40095f0 con 0x7f52c40f5410 2024-01-21T11:22:07.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.358+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c4045ad0 con 0x7f52c40f5410 2024-01-21T11:22:07.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.358+0000 7f52ba7fc700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f52b002ff60 0x7f52b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:07.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.358+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52b40410a0 con 0x7f52c40f5410 2024-01-21T11:22:07.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.358+0000 7f52c91d8700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f52b002ff60 0x7f52b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:07.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.359+0000 7f52c91d8700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f52b002ff60 0x7f52b0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f52c00096a0 tx=0x7f52c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:07.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.363+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52b401d7f0 con 0x7f52c40f5410 2024-01-21T11:22:07.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.571+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52c40fae60 con 0x7f52c40f5410 2024-01-21T11:22:07.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.571+0000 7f52ba7fc700 1 -- 172.21.15.178:0/3875571841 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f52b4010230 con 0x7f52c40f5410 2024-01-21T11:22:07.571 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:22:07.571 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f52b002ff60 msgr2=0x7f52b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f52b002ff60 0x7f52b0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f52c00096a0 tx=0x7f52c0006b40).stop 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 msgr2=0x7f52c4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c4105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f52b4013510 tx=0x7f52b400b040).stop 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 shutdown_connections 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f52b002ff60 0x7f52b0032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 --2- 172.21.15.178:0/3875571841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40f5410 0x7f52c4105630 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 >> 172.21.15.178:0/3875571841 conn(0x7f52c40f0b70 msgr2=0x7f52c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 shutdown_connections 2024-01-21T11:22:07.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:07.573+0000 7f52cbc3d700 1 -- 172.21.15.178:0/3875571841 wait complete. 2024-01-21T11:22:07.574 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:22:08.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:07 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:22:07.572128+0000 mon.smithi089 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.178:0/3875571841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:22:08.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:22:08.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:22:09.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:08 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:22:07.722511+0000 mgr.smithi089.ufybqw (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:10.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 -- 172.21.15.178:0/4195307442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 msgr2=0x7f43100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:10.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 --2- 172.21.15.178:0/4195307442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f43100f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f4300009fb0 tx=0x7f4300009b30).stop 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 -- 172.21.15.178:0/4195307442 shutdown_connections 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 --2- 172.21.15.178:0/4195307442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f43100f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 -- 172.21.15.178:0/4195307442 >> 172.21.15.178:0/4195307442 conn(0x7f43100f0b70 msgr2=0x7f43100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 -- 172.21.15.178:0/4195307442 shutdown_connections 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.136+0000 7f4317c2b700 1 -- 172.21.15.178:0/4195307442 wait complete. 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f4317c2b700 1 Processor -- start 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f4317c2b700 1 -- start start 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f4317c2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f4310109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f4317c2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4310060240 con 0x7f43100f5410 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f43159c7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f4310109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f43159c7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f4310109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:56458/0 (socket says 172.21.15.178:56458) 2024-01-21T11:22:10.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f43159c7700 1 -- 172.21.15.178:0/2943668890 learned_addr learned my addr 172.21.15.178:0/2943668890 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f43159c7700 1 -- 172.21.15.178:0/2943668890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4300005040 con 0x7f43100f5410 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.137+0000 7f43159c7700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f4310109430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4300013510 tx=0x7f430000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.138+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4300009320 con 0x7f43100f5410 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.138+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4310060440 con 0x7f43100f5410 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.138+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4300013f00 con 0x7f43100f5410 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.138+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4300014d90 con 0x7f43100f5410 2024-01-21T11:22:10.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.138+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4310109ae0 con 0x7f43100f5410 2024-01-21T11:22:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.138+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43000095f0 con 0x7f43100f5410 2024-01-21T11:22:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.139+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310045ad0 con 0x7f43100f5410 2024-01-21T11:22:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.139+0000 7f43067fc700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42fc02ff60 0x7f42fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.139+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43000410c0 con 0x7f43100f5410 2024-01-21T11:22:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.139+0000 7f43151c6700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42fc02ff60 0x7f42fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.139+0000 7f43151c6700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42fc02ff60 0x7f42fc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f430c0096a0 tx=0x7f430c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:10.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.144+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4300014420 con 0x7f43100f5410 2024-01-21T11:22:10.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.351+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43100fae60 con 0x7f43100f5410 2024-01-21T11:22:10.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.351+0000 7f43067fc700 1 -- 172.21.15.178:0/2943668890 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f430001d7f0 con 0x7f43100f5410 2024-01-21T11:22:10.351 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:22:10.352 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42fc02ff60 msgr2=0x7f42fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42fc02ff60 0x7f42fc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f430c0096a0 tx=0x7f430c006b40).stop 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 msgr2=0x7f4310109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f4310109430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4300013510 tx=0x7f430000b040).stop 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 shutdown_connections 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7f42fc02ff60 0x7f42fc032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 --2- 172.21.15.178:0/2943668890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43100f5410 0x7f4310109430 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 >> 172.21.15.178:0/2943668890 conn(0x7f43100f0b70 msgr2=0x7f43100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 shutdown_connections 2024-01-21T11:22:10.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:10.353+0000 7f4317c2b700 1 -- 172.21.15.178:0/2943668890 wait complete. 2024-01-21T11:22:10.354 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:22:11.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:22:09.722791+0000 mgr.smithi089.ufybqw (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:11.184 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:10 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: audit 2024-01-21T11:22:10.352233+0000 mon.smithi089 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.178:0/2943668890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:22:11.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:22:11.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:22:12.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 -- 172.21.15.178:0/3021736406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 msgr2=0x7fb0e00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 --2- 172.21.15.178:0/3021736406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e00f90f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fb0d4009fb0 tx=0x7fb0d4009b30).stop 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 -- 172.21.15.178:0/3021736406 shutdown_connections 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 --2- 172.21.15.178:0/3021736406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e00f90f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 -- 172.21.15.178:0/3021736406 >> 172.21.15.178:0/3021736406 conn(0x7fb0e00f0b70 msgr2=0x7fb0e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 -- 172.21.15.178:0/3021736406 shutdown_connections 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 -- 172.21.15.178:0/3021736406 wait complete. 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.907+0000 7fb0e6884700 1 Processor -- start 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0e6884700 1 -- start start 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0e6884700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0e6884700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0106bd0 con 0x7fb0e0060280 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0dffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0dffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.178:52776/0 (socket says 172.21.15.178:52776) 2024-01-21T11:22:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0dffff700 1 -- 172.21.15.178:0/3240942463 learned_addr learned my addr 172.21.15.178:0/3240942463 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0dffff700 1 -- 172.21.15.178:0/3240942463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d4005040 con 0x7fb0e0060280 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.908+0000 7fb0dffff700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e01066e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fb0d4013510 tx=0x7fb0d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d4009320 con 0x7fb0e0060280 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e0106dd0 con 0x7fb0e0060280 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0d4013f00 con 0x7fb0e0060280 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d4014dc0 con 0x7fb0e0060280 2024-01-21T11:22:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e0107260 con 0x7fb0e0060280 2024-01-21T11:22:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0d401d3d0 con 0x7fb0e0060280 2024-01-21T11:22:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.909+0000 7fb0dcff9700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb0c802ff60 0x7fb0c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:22:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.910+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0d4041110 con 0x7fb0e0060280 2024-01-21T11:22:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.910+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e0059100 con 0x7fb0e0060280 2024-01-21T11:22:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.910+0000 7fb0df7fe700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb0c802ff60 0x7fb0c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:22:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.910+0000 7fb0df7fe700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb0c802ff60 0x7fb0c8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fb0d00096a0 tx=0x7fb0d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:22:12.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:12.914+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0d4012920 con 0x7fb0e0060280 2024-01-21T11:22:13.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.118+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0e0045ad0 con 0x7fb0e0060280 2024-01-21T11:22:13.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.119+0000 7fb0dcff9700 1 -- 172.21.15.178:0/3240942463 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb0d4012330 con 0x7fb0e0060280 2024-01-21T11:22:13.119 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-01-21T11:22:13.119 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"e44d1d4c-b84d-11ee-95b0-87774f69a715","modified":"2024-01-21T11:12:24.527360Z","created":"2024-01-21T11:12:24.527360Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:22:13.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.120+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb0c802ff60 msgr2=0x7fb0c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:13.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.120+0000 7fb0e6884700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb0c802ff60 0x7fb0c8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fb0d00096a0 tx=0x7fb0d0006b40).stop 2024-01-21T11:22:13.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.120+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 msgr2=0x7fb0e01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:22:13.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.120+0000 7fb0e6884700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e01066e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fb0d4013510 tx=0x7fb0d400b040).stop 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.121+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 shutdown_connections 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.121+0000 7fb0e6884700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:6800/111466865,v1:172.21.15.89:6801/111466865] conn(0x7fb0c802ff60 0x7fb0c8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.121+0000 7fb0e6884700 1 --2- 172.21.15.178:0/3240942463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0060280 0x7fb0e01066e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.121+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 >> 172.21.15.178:0/3240942463 conn(0x7fb0e00f0b70 msgr2=0x7fb0e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.121+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 shutdown_connections 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-01-21T11:22:13.121+0000 7fb0e6884700 1 -- 172.21.15.178:0/3240942463 wait complete. 2024-01-21T11:22:13.121 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-01-21T11:22:13.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:12 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: cluster 2024-01-21T11:22:11.723072+0000 mgr.smithi089.ufybqw (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:22:13.497 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-21T11:22:13.498 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T11:22:13.498 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T11:22:13.522 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T11:22:13.540 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T11:22:13.540 INFO:tasks.cephadm.mon.smithi089:Stopping mon.smithi089... 2024-01-21T11:22:13.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089 2024-01-21T11:22:13.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:13 smithi089 systemd[1]: Stopping Ceph mon.smithi089 for e44d1d4c-b84d-11ee-95b0-87774f69a715... 2024-01-21T11:22:13.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:22:13.674+0000 7faa777a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi089 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T11:22:13.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Jan 21 11:22:13 smithi089 ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715-mon.smithi089[38842]: debug 2024-01-21T11:22:13.674+0000 7faa777a1700 -1 mon.smithi089@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T11:22:14.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi089.service' 2024-01-21T11:22:14.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T11:22:14.254 INFO:tasks.cephadm.mon.smithi089:Stopped mon.smithi089 2024-01-21T11:22:14.254 INFO:tasks.cephadm.mon.smithi178:Stopping mon.smithi178... 2024-01-21T11:22:14.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178 2024-01-21T11:22:14.292 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178.service: Unit ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178.service not loaded. 2024-01-21T11:22:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-21T11:22:14.294 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi178" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178' 2024-01-21T11:22:14.295 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T11:22:14.297 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/remote/ubuntu@smithi089.front.sepia.ceph.com/crash 2024-01-21T11:22:14.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/crash -- . 2024-01-21T11:22:14.339 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/remote/ubuntu@smithi178.front.sepia.ceph.com/crash 2024-01-21T11:22:14.340 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/crash -- . 2024-01-21T11:22:14.382 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T11:22:14.382 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2024-01-21T11:22:14.383 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T11:22:14.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T11:22:14.412 INFO:teuthology.orchestra.run.smithi089.stderr:grep: /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T11:22:14.414 INFO:tasks.cephadm:Compressing logs... 2024-01-21T11:22:14.414 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T11:22:14.457 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T11:22:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T11:22:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:: No such file or directory 2024-01-21T11:22:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T11:22:14.482 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-mon.smithi089.log 2024-01-21T11:22:14.482 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-mgr.smithi089.ufybqw.log 2024-01-21T11:22:14.482 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-volume.log: /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-mon.smithi089.log: /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-mgr.smithi089.ufybqw.log: 90.2% -- replaced with /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T11:22:14.486 INFO:teuthology.orchestra.run.smithi089.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T11:22:14.488 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T11:22:14.488 INFO:teuthology.orchestra.run.smithi178.stderr:: No such file or directory 2024-01-21T11:22:14.490 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T11:22:14.492 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-01-21T11:22:14.492 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m0.015s 2024-01-21T11:22:14.492 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m0.013s 2024-01-21T11:22:14.492 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.012s 2024-01-21T11:22:14.558 INFO:teuthology.orchestra.run.smithi089.stderr: 92.7% -- replaced with /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-mgr.smithi089.ufybqw.log.gz 2024-01-21T11:22:14.624 INFO:teuthology.orchestra.run.smithi089.stderr: 90.6% -- replaced with /var/log/ceph/e44d1d4c-b84d-11ee-95b0-87774f69a715/ceph-mon.smithi089.log.gz 2024-01-21T11:22:14.625 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-01-21T11:22:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m0.154s 2024-01-21T11:22:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m0.225s 2024-01-21T11:22:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m0.020s 2024-01-21T11:22:14.626 INFO:tasks.cephadm:Archiving logs... 2024-01-21T11:22:14.626 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/remote/ubuntu@smithi089.front.sepia.ceph.com/log 2024-01-21T11:22:14.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T11:22:14.741 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/remote/ubuntu@smithi178.front.sepia.ceph.com/log 2024-01-21T11:22:14.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T11:22:14.775 INFO:tasks.cephadm:Removing cluster... 2024-01-21T11:22:14.776 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 --force 2024-01-21T11:24:08.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e44d1d4c-b84d-11ee-95b0-87774f69a715 --force 2024-01-21T11:24:08.369 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T11:24:08.370 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T11:24:08.389 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T11:24:08.406 INFO:tasks.cephadm:Teardown complete 2024-01-21T11:24:08.407 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178' 2024-01-21T11:24:08.632 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0f9878b41e2b401da9218a9577b70217 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178' 2024-01-21T11:24:08.637 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T11:24:08.647 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T11:24:08.676 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T11:24:08.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T11:24:08.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T11:24:08.778 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T11:24:08.789 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T11:24:08.790 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:24:08.792 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:24:08.807 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2024-01-21T11:24:08.809 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2024-01-21T11:24:08.810 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:24:08.811 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-01-21T11:24:08.811 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 -355us[ -326us] +/- 71ms 2024-01-21T11:24:08.811 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -682us[ -653us] +/- 62ms 2024-01-21T11:24:08.811 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +1190us[+1190us] +/- 82ms 2024-01-21T11:24:08.811 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:24:08.813 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:24:08.813 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-01-21T11:24:08.813 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 -312us[ -302us] +/- 71ms 2024-01-21T11:24:08.813 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -665us[ -654us] +/- 62ms 2024-01-21T11:24:08.813 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +1208us[+1208us] +/- 82ms 2024-01-21T11:24:08.813 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:24:08.814 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T11:24:08.826 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T11:24:08.827 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T11:24:08.850 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T11:24:08.911 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T11:24:08.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T11:24:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:24:09.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T11:24:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:24:09.033 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T11:24:09.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T11:24:09.071 INFO:teuthology.task.internal:Duration was 1708.618491 seconds 2024-01-21T11:24:09.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T11:24:09.102 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T11:24:09.103 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T11:24:09.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T11:24:09.150 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T11:24:09.151 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T11:24:09.445 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T11:24:09.446 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-01-21T11:24:09.446 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T11:24:09.468 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-01-21T11:24:09.468 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T11:24:09.491 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T11:24:09.491 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T11:24:09.510 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T11:24:09.580 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T11:24:09.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T11:24:09.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T11:24:09.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T11:24:09.680 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T11:24:09.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T11:24:09.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T11:24:09.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T11:24:09.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:24:09.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:24:09.774 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-01-21T11:24:09.801 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-01-21T11:24:09.828 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:24:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:24:09.865 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:24:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:24:09.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T11:24:09.897 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T11:24:09.898 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/remote/smithi089 2024-01-21T11:24:09.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T11:24:09.944 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441/remote/smithi178 2024-01-21T11:24:09.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T11:24:09.985 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T11:24:09.985 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T11:24:09.988 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T11:24:10.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T11:24:10.048 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T11:24:10.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T11:24:10.085 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T11:24:10.086 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T11:24:10.088 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T11:24:10.103 INFO:teuthology.orchestra.run.smithi089.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 11:24 /home/ubuntu/cephtest 2024-01-21T11:24:10.105 INFO:teuthology.orchestra.run.smithi178.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 11:24 /home/ubuntu/cephtest 2024-01-21T11:24:10.107 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T11:24:10.118 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T11:24:10.186 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T11:24:10.214 DEBUG:teuthology.nuke:shortname: smithi089 2024-01-21T11:24:10.214 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T11:24:10.240 DEBUG:teuthology.nuke:shortname: smithi178 2024-01-21T11:24:10.241 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T11:24:10.243 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 10:39:48.591383', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdWB1nsnue4egBQNIScTUTCvwAOSIXqEtH4of5ooKpUC2LgMPZhxI76S7i84nd+YuOCjGEXDK/d8duzbvI9woM='} 2024-01-21T11:24:10.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524441', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 10:39:48.593264', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFK1cxZMOuJXvdboXT73THFXxhxywny3PuE2Tyt3iXmqc/MURC8FjKuX8T1gMjkPk85OXlglO27FoE5++VH1Jok='} 2024-01-21T11:24:10.294 INFO:teuthology.orchestra.console.smithi089:Power off 2024-01-21T11:24:10.295 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T11:24:10.309 INFO:teuthology.orchestra.console.smithi178:Power off 2024-01-21T11:24:10.310 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T11:24:10.318 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-01-21T11:24:10.318 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:24:10.331 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-01-21T11:24:10.332 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:24:10.340 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-01-21T11:24:10.353 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-01-21T11:24:14.341 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:24:14.355 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:24:14.451 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-01-21T11:24:14.464 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-01-21T11:24:18.453 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:24:18.465 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:24:18.564 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-01-21T11:24:18.575 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-01-21T11:24:18.666 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-01-21T11:24:18.677 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-01-21T11:24:18.781 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1708.6184911727905 failure_reason: 'Command failed on smithi178 with status 5: ''sudo systemctl stop ceph-e44d1d4c-b84d-11ee-95b0-87774f69a715@mon.smithi178''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0f9878b41e2b401da9218a9577b70217 status: fail success: false 2024-01-21T11:24:18.781 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T11:24:18.867 INFO:teuthology.run:FAIL